From seth at icir.org Fri Jul 1 11:48:19 2011 From: seth at icir.org (Seth Hall) Date: Fri, 1 Jul 2011 14:48:19 -0400 Subject: [Bro-Dev] Crash in new SSL analyzer Message-ID: <0875EC55-54C6-4204-B27F-54D1C68B0233@icir.org> I'm still seeing some crashes in the new SSL analyzer and I had a question because the DPD framework seems to be causing the crash when there is a protocol violation by calling the disable_analyzer BiF. Is there something special I should be doing in the SSL analyzer to cope with that BiF being called? Right now it causes binpac to crash in some cases. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From robin at icir.org Fri Jul 1 16:22:19 2011 From: robin at icir.org (Robin Sommer) Date: Fri, 1 Jul 2011 16:22:19 -0700 Subject: [Bro-Dev] Crash in new SSL analyzer In-Reply-To: <0875EC55-54C6-4204-B27F-54D1C68B0233@icir.org> References: <0875EC55-54C6-4204-B27F-54D1C68B0233@icir.org> Message-ID: <20110701232219.GF74841@icir.org> On Fri, Jul 01, 2011 at 14:48 -0400, you wrote: > Is there something special I should be doing in the SSL analyzer to > cope with that BiF being called? Nothing specific I recall right now. If you give me a trace and point me at the code, I'll take a look. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gc355804 at ohio.edu Fri Jul 1 16:38:40 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Fri, 01 Jul 2011 16:38:40 -0700 Subject: [Bro-Dev] New logging architecture Message-ID: <4E0E5A80.5090407@ohio.edu> Hey folks: I'm working on building threading into the logging framework in parallel with the work I'm doing on DataSeries. I believe I have a plan, and was hoping to get some input from other folks on the list. At the moment, a log message: *) Is generated deep within Bro, eventually finding its way to LogMgr::Write *) In LogMgr::Write, the following happens: > Checks that an appropriate LogMgr::Stream exists for the writer > Checks that any relevant LogWriter has been properly initialized > Applies any necessary filters *) The log message is turned into a LogVal ** > In the case of a remote filter, the LogVal ** is spirited away to serializations unknown > In the case of a local filter, the LogVal ** is passed along to the appropriate LogWriter::Write for processing. So, to change this to support threading, I was planning to turn LogMgr::Stream into a self-contained object with two 0mq message-passing sockets attached: *) One write-only (PUSH) 0mq socket created by the parent LogMgr, that could be used to send messages to the Stream object (for example, something like LOG_WRITE) *) One read-only (PULL) 0mq socket created by the child Stream object, which would be used to receive the messages the LogMgr sent. After the LogMgr::Stream was created, the PUSH 0mq socket would be the only means with which to communicate with it (in order to avoid needing evil things like semaphores / condition variables). This means that the LogMgr would need to generate and pass messages to the LogMgr::Stream object. Thus, the LogWriter initialization bit and the LogWriter logging bit would both happen within the context of the Stream thread (and as a result of writing an appropriate message to the LogMgr::Stream's PUSH socket). I figure the LogMgr would need to be able to generate (at a minimum) the following types of messages: *) EnableStream *) DisableStream *) StreamInit *) StreamFinish *) RotateLog *) LogMessage (Note: as a shortcut, we could probably build a fast-track LogMessageInProc type that passed a pointer to the data to log, rather than encapsulating everything when passing within a single process... but I figure that's an optimization, and could probably be dealt with later if it proves to be necessary). Anyway, I figure that, after this point, we'd be close to having an entirely self-contained logging infrastructure; so long as the message format was standardized (and I do have a rough draft of a message format which I'd be happy to send out, assuming the above isn't too confusing and seems technically sound), anything that spoke the correct message format could act as a logger for Bro. So. . . thoughts? Does that make sense? What's bad / broken about doing things this way? Thanks, Gilbert Clark From bro at tracker.bro-ids.org Sat Jul 2 05:12:38 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 02 Jul 2011 12:12:38 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: RuluttIdejuts Message-ID: <20110702121238.3E08E36A3EE@taffy.ICSI.Berkeley.EDU> New user registration for user RuluttIdejuts -- Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Jul 2 17:37:25 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 03 Jul 2011 00:37:25 -0000 Subject: [Bro-Dev] #489: InternalWarning causing most shutdowns Message-ID: <046.707fba065aa505bef5d93d6693f2e58f@tracker.bro-ids.org> #489: InternalWarning causing most shutdowns ---------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: | ---------------------+-------------------- At the moment, most of the shutdowns I'm seeing are resulting from the InternalWarnings function. Particularly when it's called here in Val.cc: {{{ ListVal* TableVal::ConvertToPureList() const { type_list* tl = table_type->Indices()->Types(); if ( tl->length() != 1 ) InternalWarning("bad index type in TableVal::ConvertToPureList"); return ConvertToList((*tl)[0]->Tag()); } }}} Perhaps we should audit all of the places where it's being called be make sure if Bro really needs to shutdown. I get the sense in this case at least that the problem could be handled differently. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Jul 2 20:17:18 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 03 Jul 2011 03:17:18 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: xrumer Message-ID: <20110703031716.B337536A3B1@taffy.ICSI.Berkeley.EDU> New user registration for user xrumer -- Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Jul 4 09:14:51 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 04 Jul 2011 16:14:51 -0000 Subject: [Bro-Dev] #479: topic/jsiwek/pybroccoli-fixes In-Reply-To: <048.6459c0d511666c60843800ea9fb21da5@tracker.bro-ids.org> References: <048.6459c0d511666c60843800ea9fb21da5@tracker.bro-ids.org> Message-ID: <063.a04dfcfd9614bed4476bc0852718f2fd@tracker.bro-ids.org> #479: topic/jsiwek/pybroccoli-fixes -----------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by robin): * status: assigned => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Jul 4 09:17:05 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 04 Jul 2011 16:17:05 -0000 Subject: [Bro-Dev] #489: InternalWarning causing most shutdowns In-Reply-To: <046.707fba065aa505bef5d93d6693f2e58f@tracker.bro-ids.org> References: <046.707fba065aa505bef5d93d6693f2e58f@tracker.bro-ids.org> Message-ID: <061.41db9cc8be65496d1089813227680594@tracker.bro-ids.org> #489: InternalWarning causing most shutdowns ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by robin): Do you mean InternalError()? An InternalWarning() shouldn't cause any shutdowns. -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Mon Jul 4 13:45:42 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Mon, 4 Jul 2011 15:45:42 -0500 (CDT) Subject: [Bro-Dev] New logging architecture In-Reply-To: <4E0E5A80.5090407@ohio.edu> Message-ID: <349816900.513.1309812342315.JavaMail.root@zimbra-1.ncsa.uiuc.edu> > I figure the LogMgr would need to be able to generate (at a minimum) > the following types of messages: > > *) EnableStream > *) DisableStream > *) StreamInit > *) StreamFinish > *) RotateLog > *) LogMessage Does that mean the LogMgr would send messages to Stream threads even if they're disabled? Couldn't the LogMgr itself keep track of the Stream enabled/disabled state and just not send messages to disabled Streams? > anything that spoke the correct message format could act as a logger for Bro. That's a neat idea if you're hinting at something like creating Streams such that the LogMgr binds its 0MQ socket to a tcp port, allowing 0MQ sockets connected from other hosts (not necessarily running a full Bro process) to receive logs. But I think some 0MQ-specific caveats of that might be: * the same PUSH/PULL pattern may not work well because it's possible for the pusher to block on a call to zmq_send(). Maybe PUB/SUB is better when working w/ Streams that aren't inproc? * 0MQ doesn't (currently) provide a good framework for securing messages[1] or being exposed to the public Internet[2]. (I'm about to send another mail about an experiment I did with the former topic, but basically I just ended up coming to the same conclusions that they/we already talked about). - Jon [1] http://lists.zeromq.org/pipermail/zeromq-dev/2010-October/006559.html [2] http://lists.zeromq.org/pipermail/zeromq-dev/2010-September/005944.html From jsiwek at ncsa.illinois.edu Mon Jul 4 14:40:39 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Mon, 4 Jul 2011 16:40:39 -0500 (CDT) Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <854301781.515.1309812681072.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <31069991.559.1309815639169.JavaMail.root@zimbra-1.ncsa.uiuc.edu> We kind of already came to the conclusion that switching to 0MQ for the communication framework would mean dropping Bro's built-in support for SSL/TLS communication, but here's a summary of why (and how the best replacement option seems to be an external tunneling solution). With 0MQ, you're not going to have access to a fd that can be wrapped in an OpenSSL socket BIO. So my idea was to try replacing the socket BIO with memory BIOs that sit in front of the 0MQ socket and wrap/unwrap application data coming from or going onto it. Here's the code from that attempt: https://github.com/jsiwek/ZeroMQ-SSL-State-Machine And the README explains why it won't work: This was an incomplete/failed experiment in using OpenSSL as a state machine to complete a SSL/TLS handshake over a 0MQ socket. In summary, the handshake can be completed for a single connection and app. data can be exchanged over SSL/TLS for the duration of that connection, but there lies a problem in detecting when a peer is disconnected[1][2] and thus requiring a new handshake upon reconnection. In 0MQ, the 'tcp' transport is considered a "disconnected" TCP transport, meaning that the connectivity state of peers is transparent to applications. So this reaffirms previous 0MQ security discussions[3] that possible approaches are: 1) Tunneling 0MQ traffic over another channel that performs SSL in some fashion (e.g. stunnel can work). This relies on the user of the application to be able to set this up, but you get SSL/TLS strength security for "free". 2) Using a (currently non-existent) 0MQ transport implemented as some part of the core 0MQ library to encrypt hop-by-hop. If this existed, drawbacks might be that it doesn't scale well to some of 0MQ's messaging patterns and would need to be implemented differently for its supported unicast vs. multicast transports. 3) Adding a crypto layer at the application level to wrap messages with some signing + encryption before sending them across the 0MQ socket. In order for this to provide security features that SSL/TLS offers beyond minimal message authn., confidentiality, integrity, it needs to be able to use a key-exchange algorithm (possibly PAKE), and some form of MAC'd nonce (replay protection). This doesn't seem worth the risk of rolling your own, best wait until 0MQ core is taught to use a well-established protocol. [1] http://lists.zeromq.org/pipermail/zeromq-dev/2010-July/004230.html [2] http://lists.zeromq.org/pipermail/zeromq-dev/2010-August/005285.html [3] http://lists.zeromq.org/pipermail/zeromq-dev/2010-October/006562.html From gc355804 at ohio.edu Mon Jul 4 15:40:00 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Mon, 04 Jul 2011 18:40:00 -0400 Subject: [Bro-Dev] New logging architecture In-Reply-To: <349816900.513.1309812342315.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <349816900.513.1309812342315.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <4E124140.4090906@ohio.edu> On 7/4/2011 4:45 PM, Jonathan Siwek wrote: >> I figure the LogMgr would need to be able to generate (at a minimum) >> the following types of messages: >> >> *) EnableStream >> *) DisableStream >> *) StreamInit >> *) StreamFinish >> *) RotateLog >> *) LogMessage > Does that mean the LogMgr would send messages to Stream threads even if they're disabled? Couldn't the LogMgr itself keep track of the Stream enabled/disabled state and just not send messages to disabled Streams? > Good point. Yes :) Thinking about it now, I'm not sure it would benefit the client to know whether or not a stream was disabled; assuming the client has subscribed, it simply won't get any messages to log from the publisher. Logic dictates it should also be up to the client to determine when to rotate logs. So, let's drop everything but LogMessage, and enable an additional heartbeat message that gets broadcast once every 15 seconds (in the absence of a LogMessage). If the subscriber misses two consecutive heartbeats, it's safe to assume the publisher is no longer broadcasting, and the subscriber shuts down. The stream is initialized when the first LogMessage is received, and is torn down when two heartbeats are missed OR the subscriber is terminated locally (e.g. CTRL-C). >> anything that spoke the correct message format could act as a logger for Bro. > That's a neat idea if you're hinting at something like creating Streams such that the LogMgr binds its 0MQ socket to a tcp port, allowing 0MQ sockets connected from other hosts (not necessarily running a full Bro process) to receive logs. Yup. That's the idea. In the longer-term, I've been thinking about what would happen if someone tried to turn Bro into a set of well-defined, focused utilities all built around a shared protocol. It's a fun thought-experiment :) > But I think some 0MQ-specific caveats of that might be: > > * the same PUSH/PULL pattern may not work well because it's possible for the pusher to block on a call to zmq_send(). Maybe PUB/SUB is better when working w/ Streams that aren't inproc? Another good point. Is there a good reason to keep inproc streams PUSH / PULL if inter-process communications use PUB / SUB? > * 0MQ doesn't (currently) provide a good framework for securing messages[1] or being exposed to the public Internet[2]. (I'm about to send another mail about an experiment I did with the former topic, but basically I just ended up coming to the same conclusions that they/we already talked about). I wasn't really anticipating internet deployment; instead, I was thinking of limited use-cases within a Bro cluster / local environment. I think the new message scheme would be a lot more robust than what I proposed initially, but we'd still want to implement some kind of message signing (at a minimum) if we wanted to support logging across an untrusted network; if we didn't, it'd be pretty simple to flood the remote logger with a stream of invalid events. --Gilbert From robin at icir.org Mon Jul 4 15:48:51 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 4 Jul 2011 15:48:51 -0700 Subject: [Bro-Dev] New logging architecture In-Reply-To: <4E0E5A80.5090407@ohio.edu> References: <4E0E5A80.5090407@ohio.edu> Message-ID: <20110704224851.GB9414@icir.org> On Fri, Jul 01, 2011 at 16:38 -0700, you wrote: > So, to change this to support threading, I was planning to turn > LogMgr::Stream into a self-contained object with two 0mq message-passing > sockets attached: Thinking about this a bit more, I'm not sure Stream is the right piece to turn into a thread. Two issues: - That would give us only one thread per type of writer (e.g., one for all ASCII logs). - Not all the things that the Stream does are sufficiently decoupled from the main thread, and they wouldn't be thread-safe. There's in particular the filtering, where Bro goes back into script-land, but also raising the log event and just adding new filters. So how about instead turning each LogWriter into a thread in the same way as you propose: making it self-contained with message-passing 0mq sockets and a simple protocol for sending the various types of messages (like Init, Write, etc.; just as you said). > than encapsulating everything when passing within a single process... By encapsulating do you mean the LogVal::{Read,Write} serializations? I don't think we'll actually get around them. They are to make things thread-safe by decoupling the writer's data from Bro's main data structures. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gc355804 at ohio.edu Mon Jul 4 16:16:29 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Mon, 04 Jul 2011 19:16:29 -0400 Subject: [Bro-Dev] New logging architecture In-Reply-To: <20110704224851.GB9414@icir.org> References: <4E0E5A80.5090407@ohio.edu> <20110704224851.GB9414@icir.org> Message-ID: <4E1249CD.5010707@ohio.edu> > So how about instead turning each LogWriter into a thread in the same > way as you propose: making it self-contained with message-passing 0mq > sockets and a simple protocol for sending the various types of > messages (like Init, Write, etc.; just as you said). That makes sense. See note about message types, however; after reading Jon's comments, I'm starting to think that Rotate / Init / etc. should be handled by the client exclusively. Also, supporting Init, Enable, Disable, etc. would open up a world of fun security issues to deal with on the client / subscriber end. >> than encapsulating everything when passing within a single process... > By encapsulating do you mean the LogVal::{Read,Write} serializations? > I don't think we'll actually get around them. They are to make things > thread-safe by decoupling the writer's data from Bro's main data > structures. No. I was referring more to the LogWriter::Write -- when dealing with something inproc, it doesn't make sense to take the entire LogVal ** and encapsulate it into a 0mq message if we can just pass the LogVal pointer directly instead. --Gilbert From gc355804 at ohio.edu Mon Jul 4 21:14:56 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Mon, 04 Jul 2011 21:14:56 -0700 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <31069991.559.1309815639169.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <31069991.559.1309815639169.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <4E128FC0.6010203@ohio.edu> A few thoughts inline: Jonathan Siwek wrote: > We kind of already came to the conclusion that switching to 0MQ for the > communication framework would mean dropping Bro's built-in support for > SSL/TLS communication, but here's a summary of why (and how the best > replacement option seems to be an external tunneling solution). > > With 0MQ, you're not going to have access to a fd that can be wrapped in an > OpenSSL socket BIO. So my idea was to try replacing the socket BIO with > memory BIOs that sit in front of the 0MQ socket and wrap/unwrap application > data coming from or going onto it. Here's the code from that attempt: > > https://github.com/jsiwek/ZeroMQ-SSL-State-Machine > > And the README explains why it won't work: > > This was an incomplete/failed experiment in using OpenSSL as a state machine > to complete a SSL/TLS handshake over a 0MQ socket. In summary, the handshake > can be completed for a single connection and app. data can be exchanged over > SSL/TLS for the duration of that connection, but there lies a problem in > detecting when a peer is disconnected[1][2] and thus requiring a new handshake > upon reconnection. What about DTLS? I think OpenSSL supports that, but I'm not sure how well. I would see that protocol mapping more naturally to 0mq's idea of messages and / or disconnected transports, and the connection emulation it provides *might* work on top of 0mq. (haven't looked at the code yet, so apologies if that's explained within :) > 1) Tunneling 0MQ traffic over another channel that performs SSL in some fashion > (e.g. stunnel can work). This relies on the user of the application to > be able to set this up, but you get SSL/TLS strength security for "free". > One of the things I don't like about transparently tunneling / encrypting like this is that an increase in network I/O also leads to an increase in CPU utilization; thus, as the amount of traffic bro analyzes increases (and its use increases), so too does the load on the system as a whole. This kind of transparent encryption, then, could lead to some interesting behavior on older systems. I don't know how relevant the statistics are, but this thread is still interesting to read: http://openvpn.net/archive/openvpn-users/2005-06/msg00224.html Anyway, the point is that while a VPN might Just Work (tm) from an administrative / systems perspective, we'd probably need to test to see exactly how many events we could push through the VPN on various systems while simultaneously processing packets before the virtual links found themselves CPU-bound. > 3) Adding a crypto layer at the application level to wrap messages with some > signing + encryption before sending them across the 0MQ socket. In order > for this to provide security features that SSL/TLS offers beyond minimal > message authn., confidentiality, integrity, it needs to be able to use a > key-exchange algorithm (possibly PAKE), and some form of MAC'd nonce > (replay protection). This doesn't seem worth the risk of rolling your own, > best wait until 0MQ core is taught to use a well-established protocol. > +1; implementing anything more complex than signing seems like a bad idea. --Gilbert From robin at icir.org Mon Jul 4 21:31:37 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 4 Jul 2011 21:31:37 -0700 Subject: [Bro-Dev] New logging architecture In-Reply-To: <4E1249CD.5010707@ohio.edu> References: <4E0E5A80.5090407@ohio.edu> <20110704224851.GB9414@icir.org> <4E1249CD.5010707@ohio.edu> Message-ID: <20110705043137.GB17340@icir.org> On Mon, Jul 04, 2011 at 19:16 -0400, you wrote: > That makes sense. See note about message types, however; after reading > Jon's comments, I'm starting to think that Rotate / Init / etc. should > be handled by the client exclusively. Not quite sure what you mean be handled exlusively. They'd still need to be triggered by the LogManager so we would need messages to send the action over to then be performed by the client. So what I'd picture is essentially one message per method in LogWriter. One reason that the manager needs to trigger them is that there are parameters to pass (like path, etc. for Init). There're more though: some can be triggered explicitly from the script-layer at any time (e.g., Flush()); and others need to be executed synchronously across log files (rotation: all files should rotate at the same time as otherwise one gets inconsistent timestamps). > No. I was referring more to the LogWriter::Write -- when dealing with > something inproc, it doesn't make sense to take the entire LogVal ** and > encapsulate it into a 0mq message if we can just pass the LogVal pointer > directly instead. Got it. Yes, that makes sense. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Mon Jul 4 21:40:04 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 4 Jul 2011 21:40:04 -0700 Subject: [Bro-Dev] New logging architecture In-Reply-To: <4E124140.4090906@ohio.edu> References: <349816900.513.1309812342315.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <4E124140.4090906@ohio.edu> Message-ID: <20110705044004.GC17340@icir.org> On Mon, Jul 04, 2011 at 18:40 -0400, you wrote: > So, let's drop everything but LogMessage, and enable an additional > heartbeat message that gets broadcast once every 15 seconds (in the > absence of a LogMessage). If the subscriber misses two consecutive > heartbeats, it's safe to assume the publisher is no longer broadcasting, > and the subscriber shuts down. Hmmm ... So this is something that would be needed for communication to external but it wouldn't if we did just in inproc, right? Not sure I like it then. It makes the internal communication between LogMgr and LogWriter quite complex for a case that'll be pretty rarely used later. Here's an alternative suggestion: why don't we leave the LogMgr-to-LogWriter channel simple and just do inproc communication there. Then, in addition, we can create a LogForwarder class derived from LogWriter that in turn uses 0mq to talk to the outside the work. This forwarder can be as complex as needed, and just like any other writer it would run in its own thread. Does that make sense? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Mon Jul 4 21:58:53 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 4 Jul 2011 21:58:53 -0700 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <31069991.559.1309815639169.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <854301781.515.1309812681072.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <31069991.559.1309815639169.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <20110705045853.GD17340@icir.org> Thanks a lot for the summary and for trying this, even if eventually unsuccessful. On Mon, Jul 04, 2011 at 16:40 -0500, you wrote: > upon reconnection. In 0MQ, the 'tcp' transport is considered a "disconnected" > TCP transport, meaning that the connectivity state of peers is transparent > to applications. Oh, that's actually something that could bite us in another way as well. When Bro starts talking to Bro, there's some state that's exchanged initially just after the connection has been setup and before "normal" messages start being exchanged. If we don't learn about a reconnect (which is I how interpret your statement above), we can't do that state exchange. This *may* be something we could get around by changing parts of the protocol but (1) that would make switching to 0mq quite a bit more complicated, and (2) I'm not sure right now whether it would work at all. Is there a way around this, like not doing transparent reconnects and setting up new connections instead? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Tue Jul 5 04:09:52 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 05 Jul 2011 11:09:52 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: usequitle Message-ID: <20110705110952.2BB0A36A398@taffy.ICSI.Berkeley.EDU> New user registration for user usequitle -- Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Jul 5 04:46:15 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 05 Jul 2011 11:46:15 -0000 Subject: [Bro-Dev] #490: Segfault in logging framework Message-ID: <046.aea7f8c7ca241603b6858d80236ddc86@tracker.bro-ids.org> #490: Segfault in logging framework ---------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: | ---------------------+-------------------- There was no message on stderr or stdout, but here's the stack trace. {{{ Program terminated with signal 11, Segmentation fault. #0 0x0820e23a in LogWriterAscii::DoWriteOne (this=0xac22718, desc=0xbfe67f24, val=0xd5e7cb0, field=0xac22670) at /usr/local/bro/src/LogWriterAscii.cc:175 175 if ( ! DoWriteOne(desc, val->val.set_val.vals[j], field) ) (gdb) bt #0 0x0820e23a in LogWriterAscii::DoWriteOne (this=0xac22718, desc=0xbfe67f24, val=0xd5e7cb0, field=0xac22670) at /usr/local/bro/src/LogWriterAscii.cc:175 #1 0x0820e3ba in LogWriterAscii::DoWrite (this=0xac22718, num_fields=17, fields=0xabc9138, vals=0xe6bd038) at /usr/local/bro/src/LogWriterAscii.cc:222 #2 0x0820d292 in LogWriter::Write (this=0xac22718, arg_num_fields=17, vals=0xe6bd038) at /usr/local/bro/src/LogWriter.cc:60 #3 0x0820afd9 in LogMgr::Write (this=0x9ec6d50, id=0xa1a57d8, columns=0xee3d2c0) at /usr/local/bro/src/LogMgr.cc:947 #4 0x081cb8dc in BifFunc::Log::bro___write (frame=0xc9800f0, BiF_ARGS=0xdaa7190) at logging.bif:49 #5 0x081c620d in BuiltinFunc::Call (this=0xa09a7d0, args=0xdaa7190, parent=0xc9800f0) at /usr/local/bro/src/Func.cc:462 #6 0x081b39c9 in CallExpr::Eval (this=0x9f3ae78, f=0xc9800f0) at /usr/local/bro/src/Expr.cc:4651 #7 0x0826f1e1 in StmtList::Exec(Frame *, ._127 &) const (this=0x9f3ab18, f=0xc9800f0, flow=@0xbfe68244) at /usr/local/bro/src/Stmt.cc:1432 #8 0x081c6a16 in BroFunc::Call (this=0x9f3af48, args=0xc3b33a0, parent=0xe9fd878) at /usr/local/bro/src/Func.cc:319 #9 0x081b39c9 in CallExpr::Eval (this=0xa1a5a50, f=0xe9fd878) at /usr/local/bro/src/Expr.cc:4651 #10 0x082742cc in ExprStmt::Exec(Frame *, ._127 &) const (this=0xa1a5ab0, f=0xe9fd878, flow=@0xbfe68434) at /usr/local/bro/src/Stmt.cc:397 #11 0x0826f1e1 in StmtList::Exec(Frame *, ._127 &) const (this=0xa1a2ac8, f=0xe9fd878, flow=@0xbfe68434) at /usr/local/bro/src/Stmt.cc:1432 #12 0x0826e729 in IfStmt::DoExec(Frame *, Val *, ._127 &) const (this=0xa1a5b78, f=0xe9fd878, v=0xe676f20, flow=@0xbfe68434) at /usr/local/bro/src/Stmt.cc:512 #13 0x082742ec in ExprStmt::Exec(Frame *, ._127 &) const (this=0xa1a5b78, f=0xe9fd878, flow=@0xbfe68434) at /usr/local/bro/src/Stmt.cc:401 #14 0x0826f1e1 in StmtList::Exec(Frame *, ._127 &) const (this=0xa1a21e8, f=0xe9fd878, flow=@0xbfe68434) at /usr/local/bro/src/Stmt.cc:1432 #15 0x081c6a16 in BroFunc::Call (this=0xa1a61b0, args=0xd78afa8, parent=0xde10150) at /usr/local/bro/src/Func.cc:319 #16 0x081b39c9 in CallExpr::Eval (this=0xa1b67f8, f=0xde10150) at /usr/local/bro/src/Expr.cc:4651 #17 0x082742cc in ExprStmt::Exec(Frame *, ._127 &) const (this=0xa1b6858, f=0xde10150, flow=@0xbfe68624) at /usr/local/bro/src/Stmt.cc:397 #18 0x0826f1e1 in StmtList::Exec(Frame *, ._127 &) const (this=0xa1b5ec8, f=0xde10150, flow=@0xbfe68624) at /usr/local/bro/src/Stmt.cc:1432 #19 0x0826e729 in IfStmt::DoExec(Frame *, Val *, ._127 &) const (this=0xa1b6920, f=0xde10150, v=0xf1505b0, flow=@0xbfe68624) at /usr/local/bro/src/Stmt.cc:512 #20 0x082742ec in ExprStmt::Exec(Frame *, ._127 &) const (this=0xa1b6920, f=0xde10150, flow=@0xbfe68624) at /usr/local/bro/src/Stmt.cc:401 #21 0x0826f1e1 in StmtList::Exec(Frame *, ._127 &) const (this=0xa1ac7d0, f=0xde10150, flow=@0xbfe68624) at /usr/local/bro/src/Stmt.cc:1432 #22 0x081c6a16 in BroFunc::Call (this=0x9f8bd88, args=0xd5ca7c8, parent=0x0) at /usr/local/bro/src/Func.cc:319 #23 0x0817d413 in EventHandler::Call (this=0x9ed4848, vl=0xd5ca7c8, no_remote=false) at /usr/local/bro/src/EventHandler.cc:96 #24 0x0817caf7 in Dispatch (this=0x8340400) at /usr/local/bro/src/Event.h:43 #25 EventMgr::Dispatch (this=0x8340400) at /usr/local/bro/src/Event.cc:107 ---Type to continue, or q to quit--- #26 0x0817cc18 in EventMgr::Drain (this=0x8340400) at /usr/local/bro/src/Event.cc:119 #27 0x0821dd55 in net_packet_dispatch (t=1309733647.4661829, hdr=0xa7769c0, pkt=0xa776e7c "E", hdr_size=0, src_ps=0xa776988, pkt_elem=0x0) at /usr/local/bro/src/Net.cc:381 #28 0x0821e169 in net_packet_arrival (t=1309733647.4661829, hdr=0xa7769c0, pkt=0xa776e7c "E", hdr_size=0, src_ps=0xa776988) at /usr/local/bro/src/Net.cc:443 #29 0x0822d4b1 in PktSrc::Process (this=0xa776988) at /usr/local/bro/src/PktSrc.cc:275 #30 0x0821dee7 in net_run () at /usr/local/bro/src/Net.cc:473 #31 0x08136227 in main (argc=227842232, argv=0xbfe68c14) at /usr/local/bro/src/main.cc:1009 (gdb) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Jul 5 04:48:23 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 05 Jul 2011 11:48:23 -0000 Subject: [Bro-Dev] #491: Another logging framework segfault Message-ID: <046.62907b162b4bdc21b0d64b22b008e9be@tracker.bro-ids.org> #491: Another logging framework segfault ---------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: | ---------------------+-------------------- {{{ (gdb) bt #0 0x00457ae4 in free () from /lib/libc.so.6 #1 0x072c75c1 in operator delete(void*) () from /usr/lib/libstdc++.so.6 #2 0x072c761d in operator delete[](void*) () from /usr/lib/libstdc++.so.6 #3 0x0820776e in LogVal::~LogVal (this=0x1ab529e0, __in_chrg=) at /usr/local/bro/src/LogMgr.cc:101 #4 0x0820d20b in LogWriter::DeleteVals (this=0x9c9a348, vals=0x7ee62588) at /usr/local/bro/src/LogWriter.cc:147 #5 0x0820d2a3 in LogWriter::Write (this=0x9c9a348, arg_num_fields=17, vals=0x7ee62588) at /usr/local/bro/src/LogWriter.cc:62 #6 0x0820afd9 in LogMgr::Write (this=0x901fe50, id=0x92fdc80, columns=0x4b202710) at /usr/local/bro/src/LogMgr.cc:947 #7 0x081cb8dc in BifFunc::Log::bro___write (frame=0x3e74c270, BiF_ARGS=0x44a5c300) at logging.bif:49 #8 0x081c620d in BuiltinFunc::Call (this=0x91f3218, args=0x44a5c300, parent=0x3e74c270) at /usr/local/bro/src/Func.cc:462 #9 0x081b39c9 in CallExpr::Eval (this=0x9094230, f=0x3e74c270) at /usr/local/bro/src/Expr.cc:4651 #10 0x0826f1e1 in StmtList::Exec(Frame *, ._127 &) const (this=0x9093ed0, f=0x3e74c270, flow=@0xbfa42934) at /usr/local/bro/src/Stmt.cc:1432 #11 0x081c6a16 in BroFunc::Call (this=0x9094300, args=0x40c29858, parent=0x6e9e2100) at /usr/local/bro/src/Func.cc:319 #12 0x081b39c9 in CallExpr::Eval (this=0x92fdef8, f=0x6e9e2100) at /usr/local/bro/src/Expr.cc:4651 #13 0x082742cc in ExprStmt::Exec(Frame *, ._127 &) const (this=0x92fdf58, f=0x6e9e2100, flow=@0xbfa42b24) at /usr/local/bro/src/Stmt.cc:397 #14 0x0826f1e1 in StmtList::Exec(Frame *, ._127 &) const (this=0x92faf70, f=0x6e9e2100, flow=@0xbfa42b24) at /usr/local/bro/src/Stmt.cc:1432 #15 0x0826e729 in IfStmt::DoExec(Frame *, Val *, ._127 &) const (this=0x92fe020, f=0x6e9e2100, v=0x4743a370, flow=@0xbfa42b24) at /usr/local/bro/src/Stmt.cc:512 #16 0x082742ec in ExprStmt::Exec(Frame *, ._127 &) const (this=0x92fe020, f=0x6e9e2100, flow=@0xbfa42b24) at /usr/local/bro/src/Stmt.cc:401 #17 0x0826f1e1 in StmtList::Exec(Frame *, ._127 &) const (this=0x92fa690, f=0x6e9e2100, flow=@0xbfa42b24) at /usr/local/bro/src/Stmt.cc:1432 #18 0x081c6a16 in BroFunc::Call (this=0x92fe658, args=0x563a9fe0, parent=0x52d77848) at /usr/local/bro/src/Func.cc:319 #19 0x081b39c9 in CallExpr::Eval (this=0x930ec58, f=0x52d77848) at /usr/local/bro/src/Expr.cc:4651 #20 0x082742cc in ExprStmt::Exec(Frame *, ._127 &) const (this=0x930ecb8, f=0x52d77848, flow=@0xbfa42d14) at /usr/local/bro/src/Stmt.cc:397 #21 0x0826f1e1 in StmtList::Exec(Frame *, ._127 &) const (this=0x930e328, f=0x52d77848, flow=@0xbfa42d14) at /usr/local/bro/src/Stmt.cc:1432 #22 0x0826e729 in IfStmt::DoExec(Frame *, Val *, ._127 &) const (this=0x930ed80, f=0x52d77848, v=0x3f221658, flow=@0xbfa42d14) at /usr/local/bro/src/Stmt.cc:512 #23 0x082742ec in ExprStmt::Exec(Frame *, ._127 &) const (this=0x930ed80, f=0x52d77848, flow=@0xbfa42d14) at /usr/local/bro/src/Stmt.cc:401 #24 0x0826f1e1 in StmtList::Exec(Frame *, ._127 &) const (this=0x9304cc0, f=0x52d77848, flow=@0xbfa42d14) at /usr/local/bro/src/Stmt.cc:1432 #25 0x081c6a16 in BroFunc::Call (this=0x90e4ed0, args=0x3c4c9358, parent=0x0) at /usr/local/bro/src/Func.cc:319 ---Type to continue, or q to quit--- #26 0x0817d413 in EventHandler::Call (this=0x902d940, vl=0x3c4c9358, no_remote=false) at /usr/local/bro/src/EventHandler.cc:96 #27 0x0817caf7 in Dispatch (this=0x8340400) at /usr/local/bro/src/Event.h:43 #28 EventMgr::Dispatch (this=0x8340400) at /usr/local/bro/src/Event.cc:107 #29 0x0817cc18 in EventMgr::Drain (this=0x8340400) at /usr/local/bro/src/Event.cc:119 #30 0x0821dd55 in net_packet_dispatch (t=1309686477.2723601, hdr=0x98d4f00, pkt=0x98d66c4 "E\020", hdr_size=0, src_ps=0x98d4ec8, pkt_elem=0x0) at /usr/local/bro/src/Net.cc:381 #31 0x0821e169 in net_packet_arrival (t=1309686477.2723601, hdr=0x98d4f00, pkt=0x98d66c4 "E\020", hdr_size=0, src_ps=0x98d4ec8) at /usr/local/bro/src/Net.cc:443 #32 0x0822d4b1 in PktSrc::Process (this=0x98d4ec8) at /usr/local/bro/src/PktSrc.cc:275 #33 0x0821dee7 in net_run () at /usr/local/bro/src/Net.cc:473 #34 0x08136227 in main (argc=Cannot access memory at address 0x0 ) at /usr/local/bro/src/main.cc:1009 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jul 6 00:16:34 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 06 Jul 2011 07:16:34 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: TarasBulba Message-ID: <20110706071632.17D0D36A37B@taffy.ICSI.Berkeley.EDU> New user registration for user TarasBulba -- Bro Tracker Bro Issue Tracker From seth at icir.org Wed Jul 6 03:52:16 2011 From: seth at icir.org (Seth Hall) Date: Wed, 6 Jul 2011 06:52:16 -0400 Subject: [Bro-Dev] malloc problem in master Message-ID: <2C00486D-26E2-4C05-AF34-12529FDEBC46@icir.org> There is some problem with Bro in master, if you start it without any arguments it throws a malloc error. [seth at Blake build (master)]$ gdb ./src/bro (gdb) break malloc_error_break Breakpoint 1 at 0x83126e977eb499 (gdb) r Starting program: ./src/bro bro(32651) malloc: *** error for object 0x1002721b8: pointer being freed was not allocated *** set a breakpoint in malloc_error_break to debug Breakpoint 1, 0x00007fff81a1c499 in malloc_error_break () (gdb) bt #0 0x00007fff81a1c499 in malloc_error_break () #1 0x00007fff81946183 in free () #2 0x00000001000a0503 in load_files_with_prefix (orig_file=0x100b1a680 "-") at scan.l:628 #3 0x00000001000a0752 in browrap () at scan.l:813 #4 0x00000001000a4048 in brolex () at ./src/scan.cc:2496 #5 0x000000010008f10e in yyparse () at p.cc:2333 #6 0x00000001000a6bea in main (argc=1, argv=0x7fff5fbfed90) at ./src/main.cc:735 Looks like it might be something might be wrong in the load_files_with_prefix function? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Wed Jul 6 16:26:35 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 06 Jul 2011 23:26:35 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: bmqwn Message-ID: <20110706232634.968E236A00A@taffy.ICSI.Berkeley.EDU> New user registration for user bmqwn -- Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jul 6 17:21:56 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 07 Jul 2011 00:21:56 -0000 Subject: [Bro-Dev] #491: Another logging framework segfault In-Reply-To: <046.62907b162b4bdc21b0d64b22b008e9be@tracker.bro-ids.org> References: <046.62907b162b4bdc21b0d64b22b008e9be@tracker.bro-ids.org> Message-ID: <061.3f54a12d6363acd8b786560a3f0ed5fb@tracker.bro-ids.org> #491: Another logging framework segfault ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Old description: > {{{ > > (gdb) bt > #0 0x00457ae4 in free () from /lib/libc.so.6 > #1 0x072c75c1 in operator delete(void*) () from /usr/lib/libstdc++.so.6 > #2 0x072c761d in operator delete[](void*) () from > /usr/lib/libstdc++.so.6 > #3 0x0820776e in LogVal::~LogVal (this=0x1ab529e0, __in_chrg= optimized out>) at /usr/local/bro/src/LogMgr.cc:101 > #4 0x0820d20b in LogWriter::DeleteVals (this=0x9c9a348, vals=0x7ee62588) > at /usr/local/bro/src/LogWriter.cc:147 > #5 0x0820d2a3 in LogWriter::Write (this=0x9c9a348, arg_num_fields=17, > vals=0x7ee62588) at /usr/local/bro/src/LogWriter.cc:62 > #6 0x0820afd9 in LogMgr::Write (this=0x901fe50, id=0x92fdc80, > columns=0x4b202710) at /usr/local/bro/src/LogMgr.cc:947 > #7 0x081cb8dc in BifFunc::Log::bro___write (frame=0x3e74c270, > BiF_ARGS=0x44a5c300) at logging.bif:49 > #8 0x081c620d in BuiltinFunc::Call (this=0x91f3218, args=0x44a5c300, > parent=0x3e74c270) at /usr/local/bro/src/Func.cc:462 > #9 0x081b39c9 in CallExpr::Eval (this=0x9094230, f=0x3e74c270) at > /usr/local/bro/src/Expr.cc:4651 > #10 0x0826f1e1 in StmtList::Exec(Frame *, ._127 &) const (this=0x9093ed0, > f=0x3e74c270, flow=@0xbfa42934) at /usr/local/bro/src/Stmt.cc:1432 > #11 0x081c6a16 in BroFunc::Call (this=0x9094300, args=0x40c29858, > parent=0x6e9e2100) at /usr/local/bro/src/Func.cc:319 > #12 0x081b39c9 in CallExpr::Eval (this=0x92fdef8, f=0x6e9e2100) at > /usr/local/bro/src/Expr.cc:4651 > #13 0x082742cc in ExprStmt::Exec(Frame *, ._127 &) const (this=0x92fdf58, > f=0x6e9e2100, flow=@0xbfa42b24) at /usr/local/bro/src/Stmt.cc:397 > #14 0x0826f1e1 in StmtList::Exec(Frame *, ._127 &) const (this=0x92faf70, > f=0x6e9e2100, flow=@0xbfa42b24) at /usr/local/bro/src/Stmt.cc:1432 > #15 0x0826e729 in IfStmt::DoExec(Frame *, Val *, ._127 &) const > (this=0x92fe020, f=0x6e9e2100, v=0x4743a370, flow=@0xbfa42b24) at > /usr/local/bro/src/Stmt.cc:512 > #16 0x082742ec in ExprStmt::Exec(Frame *, ._127 &) const (this=0x92fe020, > f=0x6e9e2100, flow=@0xbfa42b24) at /usr/local/bro/src/Stmt.cc:401 > #17 0x0826f1e1 in StmtList::Exec(Frame *, ._127 &) const (this=0x92fa690, > f=0x6e9e2100, flow=@0xbfa42b24) at /usr/local/bro/src/Stmt.cc:1432 > #18 0x081c6a16 in BroFunc::Call (this=0x92fe658, args=0x563a9fe0, > parent=0x52d77848) at /usr/local/bro/src/Func.cc:319 > #19 0x081b39c9 in CallExpr::Eval (this=0x930ec58, f=0x52d77848) at > /usr/local/bro/src/Expr.cc:4651 > #20 0x082742cc in ExprStmt::Exec(Frame *, ._127 &) const (this=0x930ecb8, > f=0x52d77848, flow=@0xbfa42d14) at /usr/local/bro/src/Stmt.cc:397 > #21 0x0826f1e1 in StmtList::Exec(Frame *, ._127 &) const (this=0x930e328, > f=0x52d77848, flow=@0xbfa42d14) at /usr/local/bro/src/Stmt.cc:1432 > #22 0x0826e729 in IfStmt::DoExec(Frame *, Val *, ._127 &) const > (this=0x930ed80, f=0x52d77848, v=0x3f221658, flow=@0xbfa42d14) at > /usr/local/bro/src/Stmt.cc:512 > #23 0x082742ec in ExprStmt::Exec(Frame *, ._127 &) const (this=0x930ed80, > f=0x52d77848, flow=@0xbfa42d14) at /usr/local/bro/src/Stmt.cc:401 > #24 0x0826f1e1 in StmtList::Exec(Frame *, ._127 &) const (this=0x9304cc0, > f=0x52d77848, flow=@0xbfa42d14) at /usr/local/bro/src/Stmt.cc:1432 > #25 0x081c6a16 in BroFunc::Call (this=0x90e4ed0, args=0x3c4c9358, > parent=0x0) at /usr/local/bro/src/Func.cc:319 > ---Type to continue, or q to quit--- > #26 0x0817d413 in EventHandler::Call (this=0x902d940, vl=0x3c4c9358, > no_remote=false) at /usr/local/bro/src/EventHandler.cc:96 > #27 0x0817caf7 in Dispatch (this=0x8340400) at > /usr/local/bro/src/Event.h:43 > #28 EventMgr::Dispatch (this=0x8340400) at > /usr/local/bro/src/Event.cc:107 > #29 0x0817cc18 in EventMgr::Drain (this=0x8340400) at > /usr/local/bro/src/Event.cc:119 > #30 0x0821dd55 in net_packet_dispatch (t=1309686477.2723601, > hdr=0x98d4f00, pkt=0x98d66c4 "E\020", hdr_size=0, src_ps=0x98d4ec8, > pkt_elem=0x0) > at /usr/local/bro/src/Net.cc:381 > #31 0x0821e169 in net_packet_arrival (t=1309686477.2723601, > hdr=0x98d4f00, pkt=0x98d66c4 "E\020", hdr_size=0, src_ps=0x98d4ec8) at > /usr/local/bro/src/Net.cc:443 > #32 0x0822d4b1 in PktSrc::Process (this=0x98d4ec8) at > /usr/local/bro/src/PktSrc.cc:275 > #33 0x0821dee7 in net_run () at /usr/local/bro/src/Net.cc:473 > #34 0x08136227 in main (argc=Cannot access memory at address 0x0 > ) at /usr/local/bro/src/main.cc:1009 > }}} New description: {{{ #0 0x0838a18a in base_type(._93) (tag=3617152) at /usr/local/bro/src/Type.cc:1611 #1 0x0839cf5a in ListVal::ListVal(._112) (this=0x28b62e18, t=3617152) at /usr/local/bro/src/Val.cc:1475 #2 0x0839d038 in TableVal::ConvertToList(._112) const (this=0x26309e18, t=3617152) at /usr/local/bro/src/Val.cc:2253 #3 0x0839d1db in TableVal::ConvertToPureList (this=0x26309e18) at /usr/local/bro/src/Val.cc:2289 #4 0x082eaea8 in LogMgr::ValToLogVal (this=0xa09b298, val=0x26309e18, ty=0xa3887e8) at /usr/local/bro/src/LogMgr.cc:1039 #5 0x082eb209 in LogMgr::RecordToFilterVals (this=0xa09b298, stream=0xa9bc8d8, filter=0xa9c0b20, columns=0x26c8f4b8) at /usr/local/bro/src/LogMgr.cc:1102 #6 0x082ee37c in LogMgr::Write (this=0xa09b298, id=0xa39b2d8, columns=0x26c8f4b8) at /usr/local/bro/src/LogMgr.cc:938 #7 0x082a5be9 in BifFunc::Log::bro___write (frame=0x18758ae8, BiF_ARGS=0x1e93a220) at logging.bif:49 #8 0x082b6b09 in BuiltinFunc::Call (this=0xa283308, args=0x1e93a220, parent=0x18758ae8) at /usr/local/bro/src/Func.cc:463 #9 0x0827ed6b in CallExpr::Eval (this=0xa113350, f=0x18758ae8) at /usr/local/bro/src/Expr.cc:4651 #10 0x0836aa21 in ReturnStmt::Exec(Frame *, ._122 &) const (this=0xa1133b8, f=0x18758ae8, flow=@0xbfc4822c) at /usr/local/bro/src/Stmt.cc:1332 #11 0x0836c3e9 in StmtList::Exec(Frame *, ._122 &) const (this=0xa112fc0, f=0x18758ae8, flow=@0xbfc4822c) at /usr/local/bro/src/Stmt.cc:1392 #12 0x082bd5d3 in BroFunc::Call (this=0xa113430, args=0x3a965ee0, parent=0x3f9858d0) at /usr/local/bro/src/Func.cc:320 #13 0x0827ed6b in CallExpr::Eval (this=0xa39b578, f=0x3f9858d0) at /usr/local/bro/src/Expr.cc:4651 #14 0x0836b9db in ExprStmt::Exec(Frame *, ._122 &) const (this=0xa39b5e0, f=0x3f9858d0, flow=@0xbfc4843c) at /usr/local/bro/src/Stmt.cc:357 #15 0x0836c3e9 in StmtList::Exec(Frame *, ._122 &) const (this=0xa398300, f=0x3f9858d0, flow=@0xbfc4843c) at /usr/local/bro/src/Stmt.cc:1392 #16 0x08370737 in IfStmt::DoExec(Frame *, Val *, ._122 &) const (this=0xa39b6b8, f=0x3f9858d0, v=0x37db3588, flow=@0xbfc4843c) at /usr/local/bro/src/Stmt.cc:472 #17 0x0836ba0b in ExprStmt::Exec(Frame *, ._122 &) const (this=0xa39b6b8, f=0x3f9858d0, flow=@0xbfc4843c) at /usr/local/bro/src/Stmt.cc:361 #18 0x0836c3e9 in StmtList::Exec(Frame *, ._122 &) const (this=0xa397990, f=0x3f9858d0, flow=@0xbfc4843c) at /usr/local/bro/src/Stmt.cc:1392 #19 0x082bd5d3 in BroFunc::Call (this=0xa39bd58, args=0x25e1c220, parent=0x26626558) at /usr/local/bro/src/Func.cc:320 #20 0x0827ed6b in CallExpr::Eval (this=0xa3ad318, f=0x26626558) at /usr/local/bro/src/Expr.cc:4651 #21 0x0836b9db in ExprStmt::Exec(Frame *, ._122 &) const (this=0xa3ad380, f=0x26626558, flow=@0xbfc4864c) at /usr/local/bro/src/Stmt.cc:357 #22 0x0836c3e9 in StmtList::Exec(Frame *, ._122 &) const (this=0xa3ac960, f=0x26626558, flow=@0xbfc4864c) at /usr/local/bro/src/Stmt.cc:1392 #23 0x08370737 in IfStmt::DoExec(Frame *, Val *, ._122 &) const (this=0xa3ad458, f=0x26626558, v=0x37fa28a8, flow=@0xbfc4864c) at /usr/local/bro/src/Stmt.cc:472 #24 0x0836ba0b in ExprStmt::Exec(Frame *, ._122 &) const (this=0xa3ad458, f=0x26626558, flow=@0xbfc4864c) at /usr/local/bro/src/Stmt.cc:361 #25 0x0836c3e9 in StmtList::Exec(Frame *, ._122 &) const (this=0xa3a28f8, f=0x26626558, flow=@0xbfc4864c) at /usr/local/bro/src/Stmt.cc:1392 #26 0x082bd5d3 in BroFunc::Call (this=0xa167180, args=0x29e7c400, parent=0x0) at /usr/local/bro/src/Func.cc:320 #27 0x0825e70f in EventHandler::Call (this=0xa0a8d88, vl=0x29e7c400, no_remote=false) at /usr/local/bro/src/EventHandler.cc:73 ---Type to continue, or q to quit--- #28 0x081fd1ec in Event::Dispatch (this=0x3d8e9080, no_remote=false) at /usr/local/bro/src/Event.h:46 #29 0x0825e158 in EventMgr::Dispatch (this=0x84609c0) at /usr/local/bro/src/Event.cc:107 #30 0x0825e1b3 in EventMgr::Drain (this=0x84609c0) at /usr/local/bro/src/Event.cc:119 #31 0x08304fe6 in net_packet_dispatch (t=1309987975.776865, hdr=0xa9a8d90, pkt=0xa9aa524 "E\020", hdr_size=0, src_ps=0xa9a8d58, pkt_elem=0x0) at /usr/local/bro/src/Net.cc:353 #32 0x08305570 in net_packet_arrival (t=1309987975.776865, hdr=0xa9a8d90, pkt=0xa9aa524 "E\020", hdr_size=0, src_ps=0xa9a8d58) at /usr/local/bro/src/Net.cc:415 #33 0x08318c87 in PktSrc::Process (this=0xa9a8d58) at /usr/local/bro/src/PktSrc.cc:275 #34 0x08305296 in net_run () at /usr/local/bro/src/Net.cc:445 #35 0x081fb44c in main (argc=7, argv=0xbfc48dd4) at /usr/local/bro/src/main.cc:993 }}} -- Comment (by seth): I replaced the stack trace with a new one because the other was with a bogus patch applied. This is on an instance running current master with the Gilbert's patch in fastpath applied but it has been seen even before that patch was applied. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jul 6 17:22:30 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 07 Jul 2011 00:22:30 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: gclark Message-ID: <20110707002229.7E9D536A385@taffy.ICSI.Berkeley.EDU> New user registration for user gclark -- Bro Tracker Bro Issue Tracker From vern at icir.org Thu Jul 7 04:41:56 2011 From: vern at icir.org (Vern Paxson) Date: Thu, 07 Jul 2011 04:41:56 -0700 Subject: [Bro-Dev] http://tracker.bro-ids.org/time-machine Message-ID: <20110707114157.40989322D24@taffy.ICSI.Berkeley.EDU> This gets an ugly message: Error TracError: IOError: [Errno 13] Permission denied: u'/da/trac/projects/time-machine/log/trac.log' - Vern From bro at tracker.bro-ids.org Thu Jul 7 06:29:05 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 07 Jul 2011 13:29:05 -0000 Subject: [Bro-Dev] #492: Small bug in new reporter code Message-ID: <046.8bda721c6943df6332a38c8f506a2b5f@tracker.bro-ids.org> #492: Small bug in new reporter code ---------------------+------------------------ Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ In the new reporter.log it looks like the location field is set to nothing on some messages but then it is set to the literal string "" and it continues to the set to the same thing after that even for messages that previously weren't set that way. I believe this behavior is coming from the core events. {{{ 1310012863.9355 Reporter::ERROR bro wasn't compiled with IPv6 support - 1310020667.27968 Reporter::ERROR bro wasn't compiled with IPv6 support - 1310032760.46583 Reporter::ERROR bro wasn't compiled with IPv6 support - 1310032760.46583 Reporter::ERROR bro wasn't compiled with IPv6 support - 1310032833.05561 Reporter::ERROR bro wasn't compiled with IPv6 support - 1310036693.57515 Reporter::WARNING bad index type in TableVal::ConvertToPureList ({}) 1310037566.46695 Reporter::ERROR bro wasn't compiled with IPv6 support 1310037566.46695 Reporter::ERROR bro wasn't compiled with IPv6 support 1310039062.06773 Reporter::ERROR bro wasn't compiled with IPv6 support 1310040812.09854 Reporter::ERROR bro wasn't compiled with IPv6 support 1310040812.09854 Reporter::ERROR bro wasn't compiled with IPv6 support 1310041765.50901 Reporter::ERROR bro wasn't compiled with IPv6 support }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jul 7 06:54:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 07 Jul 2011 13:54:13 -0000 Subject: [Bro-Dev] #492: Small bug in new reporter code In-Reply-To: <046.8bda721c6943df6332a38c8f506a2b5f@tracker.bro-ids.org> References: <046.8bda721c6943df6332a38c8f506a2b5f@tracker.bro-ids.org> Message-ID: <061.16e46a13deabce2b19ea63541048f83e@tracker.bro-ids.org> #492: Small bug in new reporter code ----------------------+------------------------ Reporter: seth | Owner: robin Type: Problem | Status: accepted Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Changes (by robin): * owner: => robin * status: new => accepted -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Thu Jul 7 10:24:17 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Thu, 7 Jul 2011 12:24:17 -0500 (CDT) Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <4E128FC0.6010203@ohio.edu> Message-ID: <1053151564.2454.1310059457391.JavaMail.root@zimbra-1.ncsa.uiuc.edu> > What about DTLS? I think OpenSSL supports that, but I'm not sure how > well. > > I would see that protocol mapping more naturally to 0mq's idea of > messages and / or disconnected transports, and the connection > emulation it provides *might* work on top of 0mq. I didn't try, but don't think it helps. As a general scenario, let's say a client and server both complete a handshake over 0MQ (DTLS, SSL, TLS, whichever), but after a while of exchanging app. data, the client crashes. In any protocol, session resuming is supported provided that the client saves some state (session ID, master secret). We could do that (don't think we want to), but another question is how can the server know that the client will ever return? That seems to require implementing a heartbeat and DTLS seems to just rely on retransmission timers during the handshake? > (haven't looked at the code yet, so apologies if that's explained > within :) Not really, the code is just hacked together, but it's short enough to read/understand if you want to try anything. - Jon From robin at icir.org Thu Jul 7 11:06:25 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 7 Jul 2011 11:06:25 -0700 Subject: [Bro-Dev] malloc problem in master In-Reply-To: <2C00486D-26E2-4C05-AF34-12529FDEBC46@icir.org> References: <2C00486D-26E2-4C05-AF34-12529FDEBC46@icir.org> Message-ID: <20110707180625.GE66084@icir.org> On Wed, Jul 06, 2011 at 06:52 -0400, you wrote: > There is some problem with Bro in master, if you start it without any arguments it throws a malloc error. Found and fixed. Push forthcoming. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From jsiwek at ncsa.illinois.edu Thu Jul 7 11:21:35 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Thu, 7 Jul 2011 13:21:35 -0500 (CDT) Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <20110705045853.GD17340@icir.org> Message-ID: <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> > > upon reconnection. In 0MQ, the 'tcp' transport is considered a > > "disconnected" TCP transport, meaning that the connectivity state of peers is > > transparent to applications. > > Oh, that's actually something that could bite us in another way as > well. When Bro starts talking to Bro, there's some state that's > exchanged initially just after the connection has been setup and > before "normal" messages start being exchanged. If we don't learn > about a reconnect (which is I how interpret your statement above), we > can't do that state exchange. If the acceptor needs to recognize new connections, that doesn't seem well-suited to it, but if the client can say "I'm new" that could work. Also, the idea of having a listening socket and an accepted socket fd like in traditional socket APIs isn't there in 0MQ. You just have a socket bound at one endpoint that may accept connections from multiple endpoints, but anything written to the bound socket is going to be received by all connected endpoints. There's a ZMQ_PAIR that does communication between just two endpoints, but only a single peer can be connected at one time. I think we really need try to use 0MQ as a parallelism framework and not just as a networking library or else it's going to be a struggle to get things to work. > This *may* be something we could get around by changing parts of the > protocol but (1) that would make switching to 0mq quite a bit more > complicated, and (2) I'm not sure right now whether it would work at > all. In general, I'm getting the feeling that even the original idea of "let's just try replacing the socket code with 0MQ and increment upon that" isn't going to be easy, and to really take advantage of 0MQ's strengths requires some redesign. > Is there a way around this, like not doing transparent reconnects and > setting up new connections instead? Not that I saw. - Jon From jsiwek at ncsa.illinois.edu Thu Jul 7 11:34:17 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Thu, 7 Jul 2011 13:34:17 -0500 (CDT) Subject: [Bro-Dev] New logging architecture In-Reply-To: <4E124140.4090906@ohio.edu> Message-ID: <511012487.2882.1310063657397.JavaMail.root@zimbra-1.ncsa.uiuc.edu> > Another good point. Is there a good reason to keep inproc streams PUSH > / PULL if inter-process communications use PUB / SUB? The PUB socket has the potential to drop messages when a high-water mark is reached for a subscriber. That didn't seem acceptable for inproc. (BTW, I don't know exactly how HWMs are defined, I was just glancing at the "zmq_socket" man page). - Jon From gc355804 at ohio.edu Thu Jul 7 12:26:06 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Thu, 07 Jul 2011 15:26:06 -0400 Subject: [Bro-Dev] New logging architecture In-Reply-To: <511012487.2882.1310063657397.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <511012487.2882.1310063657397.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <4E16084E.3030000@ohio.edu> On 7/7/2011 2:34 PM, Jonathan Siwek wrote: >> Another good point. Is there a good reason to keep inproc streams PUSH >> / PULL if inter-process communications use PUB / SUB? > The PUB socket has the potential to drop messages when a high-water mark > is reached for a subscriber. That didn't seem acceptable for inproc. > (BTW, I don't know exactly how HWMs are defined, I was just glancing at > the "zmq_socket" man page). > > - Jon I could also see dropping messages under load as be a plus; wouldn't we want the event processing loop to keep running if whatever's receiving the inproc:// messages can't keep up? It's true that we could end up with slightly inconsistent logs that way; at the same time, I think this could do quite a bit to help the event processor keep chugging along in the event the system as a whole became overloaded. --Gilbert From robin at icir.org Thu Jul 7 12:28:48 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 7 Jul 2011 12:28:48 -0700 Subject: [Bro-Dev] New logging architecture In-Reply-To: <511012487.2882.1310063657397.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <4E124140.4090906@ohio.edu> <511012487.2882.1310063657397.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <20110707192848.GG66084@icir.org> Just to wrap this up: Gilbert and I talked about the inter-thread communication more the other day and decided to just do inproc between the LogMgr (i.e., the main thread) and LogWriters. If we want to add inter-process/node communication, we can later do a LogWriter that just forwards to extern whatever he receives. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gc355804 at ohio.edu Thu Jul 7 12:31:37 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Thu, 07 Jul 2011 15:31:37 -0400 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <1053151564.2454.1310059457391.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <1053151564.2454.1310059457391.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <4E160999.9000302@ohio.edu> > I didn't try, but don't think it helps. As a general scenario, let's > say a client and server both complete a handshake over 0MQ (DTLS, SSL, > TLS, whichever), but after a while of exchanging app. data, the client > crashes. > > In any protocol, session resuming is supported provided that the client > saves some state (session ID, master secret). We could do that (don't > think we want to), but another question is how can the server know > that the client will ever return? That seems to require implementing > a heartbeat and DTLS seems to just rely on retransmission timers during > the handshake? > Okay, that makes sense. Speaking of heartbeats though, what about implementing an application-level heartbeat and forcing the connection closed if X are missed (something like IRC's PING / PONG)? It's not optimal, but it might be a workaround in the short term (e.g. until 0mq acquires something native). --Gilbert From robin at icir.org Thu Jul 7 12:38:36 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 7 Jul 2011 12:38:36 -0700 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <20110707193836.GI66084@icir.org> On Thu, Jul 07, 2011 at 13:21 -0500, you wrote: > In general, I'm getting the feeling that even the original idea of "let's > just try replacing the socket code with 0MQ and increment upon that" isn't > going to be easy, and to really take advantage of 0MQ's strengths requires > some redesign. I'm getting that feeling as well, and I'm starting to wonder whether 0mq is the right tool for us at all. I'd really like to replace just the socket code with something more robust initially. We may overhaul the whole serialization (with its caching, lack of support for broadcasts, etc.) at some point as well but I think that's orthogonal and better done later/separately. Does anybody know other options for the communication layer? Ideally, it would be *C* library so Broccoli can use it directly as well. And: one conceptal change that we might consider is having Bro itslef actually use Broccoli and then handle all the communication in there. Rovin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gc355804 at ohio.edu Thu Jul 7 12:45:14 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Thu, 07 Jul 2011 15:45:14 -0400 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <4E160CCA.3050202@ohio.edu> FWIW, > I think we really need try to use 0MQ as a > parallelism framework and not just as a networking library or else it's > going to be a struggle to get things to work. +1 this. Additionally, any kind of incremental deployment within Bro's existing communication framework seems like it would be challenging. > In general, I'm getting the feeling that even the original idea of "let's > just try replacing the socket code with 0MQ and increment upon that" isn't > going to be easy, and to really take advantage of 0MQ's strengths requires > some redesign. +1 this, too. --Gilbert From seth at icir.org Thu Jul 7 12:49:32 2011 From: seth at icir.org (Seth Hall) Date: Thu, 7 Jul 2011 15:49:32 -0400 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <20110707193836.GI66084@icir.org> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> Message-ID: <76867AAA-1382-403F-B55C-D7055EA847C4@icir.org> On Jul 7, 2011, at 3:38 PM, Robin Sommer wrote: > Does anybody know other options for the communication layer? Ideally, > it would be *C* library so Broccoli can use it directly as well. Something licensed BSD would be cool too. It always bothered me that 0MQ is lesser GPL. > And: one conceptal change that we might consider is having Bro itslef > actually use Broccoli and then handle all the communication in there. I actually sort of like this idea in a way. It would result in better maintenance for broccoli and new features. I've always had this feeling like I'd be able to do something super awesome if I could have broccoli hold open a socket and let a Bro connect. Right now you can't even have broccoli->broccoli because broccoli can't be the server. > Rovin Oh, this must be a new nickname? :P .Sith -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From gc355804 at ohio.edu Thu Jul 7 13:21:46 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Thu, 07 Jul 2011 16:21:46 -0400 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <20110707193836.GI66084@icir.org> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> Message-ID: <4E16155A.9050102@ohio.edu> Re: alternate libraries, only thing I know of beyond boost::asio is ACE, but I never understood the license well enough to feel comfortable using it in any reasonably large project: http://www1.cse.wustl.edu/~schmidt/ACE-copying.html It's a slick library, but only supports C++ to the best of my knowledge. > We may overhaul > the whole serialization (with its caching, lack of support for > broadcasts, etc.) at some point as well but I think that's orthogonal > and better done later/separately. I don't necessarily agree that overhauling serialization is orthogonal, per se, to deploying a new communication framework. Instead, I feel like the serialization framework is way too tightly coupled to the communication framework, and that it's really limiting what we're able to do with communication. Figuring out what we want to do with both before we find a replacement for the communication library might be a good thing, since it would give us an idea of what kind of band-aid we would want to use while we're waiting for the serialization stuff to get redone. > And: one conceptal change that we might consider is having Bro itslef > actually use Broccoli and then handle all the communication in there. I really like this idea. I have a feeling it would enforce better design, would lead to awesome (and, more importantly, transparent) support for parallel / remote processing, and would allow the development of stuff for bro without dragging the entire bro project along as a dependency. --Gilbert From seth at icir.org Thu Jul 7 13:26:11 2011 From: seth at icir.org (Seth Hall) Date: Thu, 7 Jul 2011 16:26:11 -0400 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <4E16155A.9050102@ohio.edu> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> Message-ID: <95CD9DE7-9D93-42E5-A92B-095E43E40714@icir.org> On Jul 7, 2011, at 4:21 PM, Gilbert Clark wrote: >> And: one conceptal change that we might consider is having Bro itslef >> actually use Broccoli and then handle all the communication in there. > > I really like this idea. I have a feeling it would enforce better > design, would lead to awesome (and, more importantly, transparent) > support for parallel / remote processing, and would allow the > development of stuff for bro without dragging the entire bro project > along as a dependency. Hmm... that's already there. Broccoli doesn't depend on Bro. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From gc355804 at ohio.edu Thu Jul 7 13:32:44 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Thu, 07 Jul 2011 16:32:44 -0400 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <95CD9DE7-9D93-42E5-A92B-095E43E40714@icir.org> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <95CD9DE7-9D93-42E5-A92B-095E43E40714@icir.org> Message-ID: <4E1617EC.5050500@ohio.edu> > Hmm... that's already there. Broccoli doesn't depend on Bro. Mea culpa. I just don't know how to read configure's output, apparently :) --Gilbert From robin at icir.org Thu Jul 7 14:43:03 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 7 Jul 2011 14:43:03 -0700 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <4E16155A.9050102@ohio.edu> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> Message-ID: <20110707214303.GD77684@icir.org> On Thu, Jul 07, 2011 at 16:21 -0400, you wrote: > I don't necessarily agree that overhauling serialization is orthogonal, > per se, to deploying a new communication framework. Instead, I feel > like the serialization framework is way too tightly coupled to the > communication framework, and that it's really limiting what we're able > to do with communication. We should of course consider what communication primitives we'll need later for a new serialization framework. But other than that, it's actually not that tightly coupled. For me it's just shuffling data across a network connection. :-) Anyway, I think the result of this discussion is "back to the drawing board". That's fine, the plan was to explore 0mq whether it's suitable for us, and it looks like at best we're unsure whether it is and at worst it just is not. So I'd say we postpone this discussion to a more general post-1.6 strategy posting. If anybody feels like it, we can in the meantime collect ideas and options on a web page. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Thu Jul 7 14:51:33 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 7 Jul 2011 14:51:33 -0700 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <20110707214303.GD77684@icir.org> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> Message-ID: <20110707215133.GF77684@icir.org> On Thu, Jul 07, 2011 at 14:43 -0700, I wrote: > So I'd say we postpone this discussion to a more general post-1.6 Gilbert, the next question then is however if we really want to use 0mq for the inter-thread communication. That idea was based on the assumption that we'd be using it anyway, which doesn't necessarily seem to be the case anymore. Should we just bite the bullet and do a pure pthreads implementation? We could encapsulate the thread management into a class ThreadMgr that would take care of starting/stopping/etc. threads, and a new class Thread would be the top-level class for LogWriters to be derived from. Robin PS: Yeah, I know, C++0x will make everything better ... And I'm still very reluctant to pull Boost in ... -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gc355804 at ohio.edu Thu Jul 7 16:04:17 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Thu, 07 Jul 2011 19:04:17 -0400 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <20110707215133.GF77684@icir.org> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> <20110707215133.GF77684@icir.org> Message-ID: <4E163B71.6050804@ohio.edu> On 7/7/2011 5:51 PM, Robin Sommer wrote: > Gilbert, the next question then is however if we really want to use > 0mq for the inter-thread communication. That idea was based on the > assumption that we'd be using it anyway, which doesn't necessarily > seem to be the case anymore. *nod* Yeah. I'll be very sad to see the lock-free synchronization go. I'll additionally hate to lose the constraints 0mq enforces on design; there's a little bit of coding around the library to do, but in this case I think it'd lead to a cleaner overall design than we may get otherwise. I don't know whether the above necessarily justifies introduces the additional dependency, though, especially when I'm not sure exactly how much of a win the lock-free stuff would actually turn out to be. > Should we just bite the bullet and do a pure pthreads implementation? > We could encapsulate the thread management into a class ThreadMgr that > would take care of starting/stopping/etc. threads, and a new class > Thread would be the top-level class for LogWriters to be derived from. 0mq doesn't completely eliminate the need for threads; we still need the LogWriter running in its own context. Thus, building a thread manager is definitely on the to-do list. First, though, I thought I'd look around for a library that does something like that. If I can't find one, then it'll be time to build one. > Robin > > PS: Yeah, I know, C++0x will make everything better ... And I'm still > very reluctant to pull Boost in ... Python would make everything better. C++0x will just make things more complicated. /cynical Also, the dependencies involved with boost do suck, but pushing maintenance onto someone else can be a huge win. --Gilbert From robin at icir.org Fri Jul 8 06:22:28 2011 From: robin at icir.org (Robin Sommer) Date: Fri, 8 Jul 2011 06:22:28 -0700 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <4E163B71.6050804@ohio.edu> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> <20110707215133.GF77684@icir.org> <4E163B71.6050804@ohio.edu> Message-ID: <20110708132228.GA84600@icir.org> On Thu, Jul 07, 2011 at 19:04 -0400, you wrote: > First, though, I thought I'd look around for a library that does > something like that. If I can't find one, then it'll be time to build > one. I forgot the obvious one yesterday: Intel's TBB. That's what the multi-core Bro prototype is already using, and it's main thread abstraction is (almost?) compatible to C++0x. I could live that dependency. And it has lock-free data structures as well. I think that's actually the best option I see right now. > Also, the dependencies involved with boost do suck, but pushing > maintenance onto someone else can be a huge win. But somebody has maintain the code that's *using* Boost ... My main concern is actually that once we have Boost, folks will immediately start using pretty much any feature it provides. :-) Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gregor at icir.org Fri Jul 8 08:21:11 2011 From: gregor at icir.org (Gregor Maier) Date: Fri, 08 Jul 2011 17:21:11 +0200 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <20110708132228.GA84600@icir.org> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> <20110707215133.GF77684@icir.org> <4E163B71.6050804@ohio.edu> <20110708132228.GA84600@icir.org> Message-ID: <4E172067.8040207@icir.org> >> First, though, I thought I'd look around for a library that does >> something like that. If I can't find one, then it'll be time to build >> one. > > I forgot the obvious one yesterday: Intel's TBB. That's what the > multi-core Bro prototype is already using, and it's main thread > abstraction is (almost?) compatible to C++0x. I could live that > dependency. And it has lock-free data structures as well. I think > that's actually the best option I see right now. I would definitely use it then. Once Bro goes multi-core I think we really don't want to have to deal with two different threading libraries. And while I don't know TBB pthreads definitely is very bare-bone. However, it appears that TBB is GPLv2 with runtime exception (and not Lesser GPL). Just wondering whether that's going to be a issue for us.... See: http://gcc.gnu.org/onlinedocs/libstdc++/manual/bk01pt01ch01s02.html cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From gc355804 at ohio.edu Fri Jul 8 11:01:16 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Fri, 08 Jul 2011 14:01:16 -0400 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <20110708132228.GA84600@icir.org> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> <20110707215133.GF77684@icir.org> <4E163B71.6050804@ohio.edu> <20110708132228.GA84600@icir.org> Message-ID: <4E1745EC.7060903@ohio.edu> On 7/8/2011 9:22 AM, Robin Sommer wrote: > On Thu, Jul 07, 2011 at 19:04 -0400, you wrote: > >> First, though, I thought I'd look around for a library that does >> something like that. If I can't find one, then it'll be time to build >> one. > I forgot the obvious one yesterday: Intel's TBB. That's what the > multi-core Bro prototype is already using, and it's main thread > abstraction is (almost?) compatible to C++0x. I could live that > dependency. And it has lock-free data structures as well. I think > that's actually the best option I see right now. Looks interesting, but one thing from the FAQ that bothers me: "I write software of . Is TBB use appropriate for me? It depends on what your application profile is. TBB does not try to replace I/O threads or GUI threads or general Win Threads. TBB is best for computational tasks that are not prone to frequent waiting for I/O or events in order to proceed (this is an area the TBB team does want to tackle later). " Also, the license (GPLv2 with a linking exception) is probably sub-optimal. It could definitely be worse, though (read: bdb and Sleepycat). > But somebody has maintain the code that's *using* Boost Well, sure. That's true of any library, though, so I'm not sure I really understand this argument :) > ... My main > concern is actually that once we have Boost, folks will immediately > start using pretty much any feature it provides. :-) Okay, I'll bite :) I like the accepted answer here: http://stackoverflow.com/questions/1226206/is-there-a-reason-to-not-use-boost As long as we kept our design focused, I think we'd be fine. --Gilbert From gc355804 at ohio.edu Fri Jul 8 11:04:31 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Fri, 08 Jul 2011 14:04:31 -0400 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <4E172067.8040207@icir.org> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> <20110707215133.GF77684@icir.org> <4E163B71.6050804@ohio.edu> <20110708132228.GA84600@icir.org> <4E172067.8040207@icir.org> Message-ID: <4E1746AF.5030202@ohio.edu> > Once Bro goes multi-core I think we > really don't want to have to deal with two different threading > libraries. +1 > However, it appears that TBB is GPLv2 with runtime exception (and not > Lesser GPL). Just wondering whether that's going to be a issue for us.... > See: http://gcc.gnu.org/onlinedocs/libstdc++/manual/bk01pt01ch01s02.html > Yeah, that. Any idea what the difference is between that and LGPL version 2? --Gilbert From robin at icir.org Sat Jul 9 11:13:21 2011 From: robin at icir.org (Robin Sommer) Date: Sat, 9 Jul 2011 11:13:21 -0700 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <4E1745EC.7060903@ohio.edu> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> <20110707215133.GF77684@icir.org> <4E163B71.6050804@ohio.edu> <20110708132228.GA84600@icir.org> <4E1745EC.7060903@ohio.edu> Message-ID: <20110709181321.GG57858@icir.org> On Fri, Jul 08, 2011 at 14:01 -0400, you wrote: > It depends on what your application profile is. TBB does not try to > replace I/O threads or GUI threads or general Win Threads. I think this concerns primarily their main usage model where one uses TBB to automatically schedule the work across threads. We wouldn't need that part, and instead use it more as a portable thread abstraction. In that sense, I don't see why TBB wouldn't be a fit for what we need. The licensing, yeah, not ideal, though it would work I think. But anyway, mulling over this a bit more, let's just go with pthreads. I don't really see that introducing a new dependency is worth it just for doing logging; and that's all we need right now. Doing the mgr-to-writer communication shouldn't be that difficult with pthreads either, and while lock-free queues are nice, I'm not convinced they are crucial here. Let's make sure to encapsulate all this nicely at some well-defined places, and we can always change the implementation later. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gc355804 at ohio.edu Sat Jul 9 12:22:13 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Sat, 09 Jul 2011 15:22:13 -0400 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <20110709181321.GG57858@icir.org> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> <20110707215133.GF77684@icir.org> <4E163B71.6050804@ohio.edu> <20110708132228.GA84600@icir.org> <4E1745EC.7060903@ohio.edu> <20110709181321.GG57858@icir.org> Message-ID: <4E18AA65.7050104@ohio.edu> > But anyway, mulling over this a bit more, let's just go with pthreads. Will do. --Gilbert From gregor at icir.org Sun Jul 10 04:26:09 2011 From: gregor at icir.org (Gregor Maier) Date: Sun, 10 Jul 2011 13:26:09 +0200 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <4E18AA65.7050104@ohio.edu> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> <20110707215133.GF77684@icir.org> <4E163B71.6050804@ohio.edu> <20110708132228.GA84600@icir.org> <4E1745EC.7060903@ohio.edu> <20110709181321.GG57858@icir.org> <4E18AA65.7050104@ohio.edu> Message-ID: <4E198C51.3010404@icir.org> On 7/9/11 21:22 , Gilbert Clark wrote: > >> But anyway, mulling over this a bit more, let's just go with pthreads. > > Will do. JFYI, pthread locking/unlocking can be expensive (my impression from using them for the TimeMachine). So, doing processing stuff in (small) batches while holding the lock might be a good idea to help performance. (Then, for the Log framework the number of messages being passed is probably also not that high compared to the TM) cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Sun Jul 10 19:13:17 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 11 Jul 2011 02:13:17 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: stewhedyposte Message-ID: <20110711021315.7013B36A38C@taffy.ICSI.Berkeley.EDU> New user registration for user stewhedyposte -- Bro Tracker Bro Issue Tracker From gc355804 at ohio.edu Sun Jul 10 19:19:46 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Sun, 10 Jul 2011 22:19:46 -0400 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <4E198C51.3010404@icir.org> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> <20110707215133.GF77684@icir.org> <4E163B71.6050804@ohio.edu> <20110708132228.GA84600@icir.org> <4E1745EC.7060903@ohio.edu> <20110709181321.GG57858@icir.org> <4E18AA65.7050104@ohio.edu> <4E198C51.3010404@icir.org> Message-ID: <4E1A5DC2.9060205@ohio.edu> On 7/10/2011 7:26 AM, Gregor Maier wrote: > On 7/9/11 21:22 , Gilbert Clark wrote: >>> But anyway, mulling over this a bit more, let's just go with pthreads. >> Will do. > JFYI, pthread locking/unlocking can be expensive (my impression from > using them for the TimeMachine). So, doing processing stuff in (small) > batches while holding the lock might be a good idea to help performance. > (Then, for the Log framework the number of messages being passed is > probably also not that high compared to the TM) > Thanks for the note; will keep that in mind. --Gilbert From seth at icir.org Mon Jul 11 09:13:57 2011 From: seth at icir.org (Seth Hall) Date: Mon, 11 Jul 2011 12:13:57 -0400 Subject: [Bro-Dev] cluster framework Message-ID: <31FA4C51-BA95-4816-8090-99321F21EA7C@icir.org> I just realized that something isn't terribly obvious from the cluster framework. I didn't give an example Cluster::nodes configuration anywhere, so here is one that I was testing things with. It's likely fairly similar to the one that I will be using for the cluster tests when I manage to get those done. redef Cluster::nodes = { ["manager-1"] = [$node_type=Cluster::MANAGER, $ip=127.0.0.1, $p=47757/tcp, $workers=set("worker-1")], ["proxy-1"] = [$node_type=Cluster::PROXY, $ip=127.0.0.1, $p=47758/tcp, $manager="manager-1", $workers=set("worker-1")], ["worker-1"] = [$node_type=Cluster::WORKER, $ip=127.0.0.1, $p=47759/tcp, $proxy="proxy-1", $manager="manager-1", $interface="en0"], ["control"] = [$node_type=Cluster::CONTROL, $ip=127.0.0.1, $p=47761/tcp], ["time-machine"] = [$node_type=Cluster::TIME_MACHINE, $ip=127.0.0.1, $p=47762/tcp], }; A couple of comments too... - It uses this funny flat structure because deep structures are currently difficult and a bit messy in Bro scripts. Unfortunately this leads to the less-than-optimal pointing with $workers, $proxy, and $manager but I think it's fairly inconsequential since most people will be autogenerating this with tools like broctl. - I segmented the notion of the manager node and where "control" is performed from which is usually done by processes using broccoli in the cluster context. This should help with the future move to deep clustering where there may be multiple managers in clusters (maybe manager won't even be the correct term?). - Time machine nodes were merged directly into the cluster configuration. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Mon Jul 11 10:10:39 2011 From: seth at icir.org (Seth Hall) Date: Mon, 11 Jul 2011 13:10:39 -0400 Subject: [Bro-Dev] cluster framework In-Reply-To: <31FA4C51-BA95-4816-8090-99321F21EA7C@icir.org> References: <31FA4C51-BA95-4816-8090-99321F21EA7C@icir.org> Message-ID: On Jul 11, 2011, at 12:13 PM, Seth Hall wrote > redef Cluster::nodes = { Oh, I also forgot to mention that this needs to be defined in a script named cluster-layout.bro which needs to be located somewhere in the BROPATH. It will be automatically loaded if the cluster framework is loaded and Bro is run with the CLUSTER_NODE environment variable set to a node name in the Cluster::nodes variable. For example: CLUSTER_NODE="worker-1" bro -ieth0 frameworks/cluster .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From hlin33 at illinois.edu Mon Jul 11 22:10:14 2011 From: hlin33 at illinois.edu (Hui Lin) Date: Mon, 11 Jul 2011 22:10:14 -0700 Subject: [Bro-Dev] Hui Lin_Some question on Binpac Message-ID: Hi, I have been working on modbus protocol analyzer by Binpac, which is an application-level protocol used in SCADA network. The protocol itself is not complex and it is not hard to express it in Binpac. However, this protocol in application level does not contain the information which can tell whether the parsing packet is a "request" or a "response". For example, in HTTP protocol, the http header will include such information. But for this protocol, the headers for both the request and response are the same. So if the traffic dump that I am using for the testing starts with the request, then the analyzer can properly parse it. However, if the traffic dump starts with the response, then the analyzer will not be properly parsing the packet. So I am wondering how to resolve this problem in my protocol analyzer. Best, Hui -- Hui Lin Research Assistant DEPEND Research Group, ECE Department University of Illinois at Urbana-Champaign hlin33 at illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20110711/cc6684e9/attachment.html From robin at icir.org Tue Jul 12 08:09:54 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 12 Jul 2011 08:09:54 -0700 Subject: [Bro-Dev] Hui Lin_Some question on Binpac In-Reply-To: References: Message-ID: <20110712150954.GB79789@icir.org> On Mon, Jul 11, 2011 at 22:10 -0700, you wrote: > tell whether the parsing packet is a "request" or a "response". For example, > in HTTP protocol, the http header will include such information. While that's right, BinPAC/Bro is actually not using that information to decide which side it's parsing. Generally, Bro always considers the originator of a connection to be the client side[1], and then it drives BinPAC parsing accordingly. Take a look at http-procotol.pac, there's this piece: type HTTP_PDU(is_orig: bool) = case is_orig of { true -> request: HTTP_Request; false -> reply: HTTP_Reply; }; And in http-analyzer.pac, there is: connection HTTP_Conn(bro_analyzer: BroAnalyzer) { upflow = HTTP_Flow(true); downflow = HTTP_Flow(false); }; flow HTTP_Flow(is_orig: bool) { flowunit = HTTP_PDU(is_orig) withcontext (connection, this); ... } When Bro passes in data, it tells BinPAC which of these flows to use. >From HTTP-binpac.cc: void HTTP_Analyzer_binpac::DeliverStream(int len, const u_char* data, bool orig) { .... // orig is true here if it's originator-side data. interp->NewData(orig, data, data + len); } There's a skeleton Bro analyzer on the web page that's based on binpac. If you us that skeleton, you should pretty much automatically get this direction stuff right. See http://www.bro-ids.org/development/binpac-sample-analyzer.html Robin [1] With a few additional heuristics for corner-cases. -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Tue Jul 12 10:23:11 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 12 Jul 2011 17:23:11 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: oubimvkey Message-ID: <20110712172311.3B73436A388@taffy.ICSI.Berkeley.EDU> New user registration for user oubimvkey -- Bro Tracker Bro Issue Tracker From seth at icir.org Tue Jul 12 11:45:49 2011 From: seth at icir.org (Seth Hall) Date: Tue, 12 Jul 2011 14:45:49 -0400 Subject: [Bro-Dev] Bro script organization Message-ID: <49F719F6-9D49-4048-BEA1-B63922A87965@icir.org> This is my proposed organization standard for the Bro policy scripts since we are now breaking them down into directories by their functionality and use. I'd like to get feedback on if I've missed any areas of functionality or something should be made clearer. Once we reach some consensus I'll be documenting this on a web page and I'll go back through the existing new scripts and make sure I've followed all of the rules. :) Here are the guidelines.... frameworks/ - These wrap up reusable chunks of functionality. (i.e. notice, cluster, communication, logging, metrics, etc) - Can load other frameworks and utils. - Never load protocol specific scripts. Something is probably designed wrong if you feel like you need to load a protocol analysis script in here and you should probably try to find a way to make the integration happen the other direction. integration/ - Meant for providing the code to integrate with external utilities for sharing data and analysis in either direction. - barnyard2 integration is currently the only example for this. - Can load utils and frameworks - In rare cases, can load protocol analyzers (think of using the syslog analyzer to actually collect logs at runtime) - Yes, Bro is now a syslog server in a very real sense. :) I need to write a script that really shows this off and the analyzer needs improved a bit (i.e. TCP support), but I think that we could even carry this over and run clustered syslog setups with broctl. protocols/ - These are scripts that do protocols analysis by handling protocol specific core generated events and most of them build some amount of reusable state in the connection record. - Can load utils and frameworks. - Ok to load other protocol specific scripts if needed for analysis of the protocol. misc/ - Mostly for non-protocol analysis related functionality that may be desired but it's not meant to be reusable from a programming perspective (i.e. loaded-scripts.bro that logs which scripts were loaded). - Can only load utils and frameworks. - Not to be loaded by other scripts. tuning/ - only doing redefs. - Don't @load anything. Dependencies should already be loaded. utils/ - These are all utility functions that are nice to have in many cases and can be used in a number of ways. - Only functions. - Can only load other things in utils/ Thanks! .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From jsiwek at ncsa.illinois.edu Tue Jul 12 12:47:29 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Tue, 12 Jul 2011 14:47:29 -0500 (CDT) Subject: [Bro-Dev] Bro script organization In-Reply-To: <49F719F6-9D49-4048-BEA1-B63922A87965@icir.org> Message-ID: <4c049f66-9692-4adc-9bc9-c6d9a1e4ae53@tangent.ncsa.illinois.edu> > This is my proposed organization standard for the Bro policy scripts There's currently a support/ directory that looks like it corresponds to misc/ in the plan. Just checking whether that's the case. - Jon From seth at icir.org Tue Jul 12 13:01:42 2011 From: seth at icir.org (Seth Hall) Date: Tue, 12 Jul 2011 16:01:42 -0400 Subject: [Bro-Dev] Bro script organization In-Reply-To: <4c049f66-9692-4adc-9bc9-c6d9a1e4ae53@tangent.ncsa.illinois.edu> References: <4c049f66-9692-4adc-9bc9-c6d9a1e4ae53@tangent.ncsa.illinois.edu> Message-ID: On Jul 12, 2011, at 3:47 PM, Jonathan Siwek wrote: > There's currently a support/ directory that looks like it corresponds > to misc/ in the plan. Just checking whether that's the case. I renamed it today. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From jsiwek at ncsa.illinois.edu Tue Jul 12 15:26:24 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Tue, 12 Jul 2011 17:26:24 -0500 (CDT) Subject: [Bro-Dev] script loading changes In-Reply-To: <3bc0f5e2-79da-4ae9-b7de-c22ea7ca19c2@tangent.ncsa.illinois.edu> Message-ID: <1ffe95af-80c7-4642-8a5d-9a02154e90cb@tangent.ncsa.illinois.edu> Robin, Seth, and I were talking about doing the following changes/additions to the script loading functionality in case anyone has input: 1) When an @load or command line argument relies on BROPATH to search for the script/package, directory separator characters (i.e. '/') must be replaced with dots ('.'). e.g. "@load frameworks.software". ".bro" file extensions are still optional. This makes loading packaged scripts look more similar to other languages and better supports the new @prefixes implementation described below. 2) Allow @load to recognize absolute paths or paths relative to the currently- loading script. e.g. if foo.bro and bar.bro live in the same directory, bar.bro can do "@load ./foo". ".bro" file extension is still optional. When a script is loaded relatively, we're still going to have to be able to track where it is within BROPATH (if it is at all) in order for change (4) described below to work. This doesn't really conflict with rule (1) about replacing path separator characters with dots when naming scripts-to-load since in this case we're not relying on BROPATH, but I wonder if that's not really clear since @load is used in both cases, with (2) being implied by arguments that start with '.' or '/'. Maybe we should use a new directive other than @load for this case? 3) A new "@add x when y ..." directive that's an alternative to @load, and can be used to conditionally load script named 'x' only when the script named 'y' has already been loaded. The evaluation of that condition is postponed until after all scripts specified via @load or as command line arguments have been loaded. 'x' and 'y' can be named according to either (1) or (2) above. 4) The way the @prefixes directive works is going to change. It currently augments future @loads by additionally checking for a prefixed version of the argument, meaning @prefixes has to be specified very early on for it to work right. The new implementation of @prefixes will mean: after all scripts have been loaded, for each script and for each prefix, search BROPATH for a prefixed version of the script in the "dot-syntax" specified by (1) and load it if it exists (if the script is a __load__.bro the __load__ part is ignored in the canonicalized version of the name such that it appears as if it's a "package" once again). e.g. doing "@load frameworks.software" and a "@prefixes local" means a file in BROPATH named "local.frameworks.software.bro" in BROPATH, will be loaded at the end of the script loading process if it exists. ".bro" extension still optional. - Jon From robin at icir.org Tue Jul 12 17:40:21 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 12 Jul 2011 17:40:21 -0700 Subject: [Bro-Dev] script loading changes In-Reply-To: <1ffe95af-80c7-4642-8a5d-9a02154e90cb@tangent.ncsa.illinois.edu> References: <3bc0f5e2-79da-4ae9-b7de-c22ea7ca19c2@tangent.ncsa.illinois.edu> <1ffe95af-80c7-4642-8a5d-9a02154e90cb@tangent.ncsa.illinois.edu> Message-ID: <20110713004021.GD1091@icir.org> Nice summary, thanks! Two additional thoughts: On Tue, Jul 12, 2011 at 17:26 -0500, you wrote: > 1) When an @load or command line argument relies on BROPATH to search for > the script/package, directory separator characters (i.e. '/') must be > replaced with dots ('.'). Do we want to say "may" instead of "must"? So both versions would be acceptable. If we @load a.b.c and don't find that file literally, we try a/b/c. I think that would help help with (2) as well. > 3) A new "@add x when y ..." directive that's an alternative to @load, and > can be used to conditionally load script named 'x' only when the script > named 'y' has already been loaded. The evaluation of that condition is > postponed until after all scripts specified via @load or as command line > arguments have been loaded. 'x' and 'y' can be named according to either > (1) or (2) above. One thing we haven't considered yet: what if "x" itself has new @loads or @adds? Can we make that well-defined? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gc355804 at ohio.edu Tue Jul 12 20:00:36 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Tue, 12 Jul 2011 23:00:36 -0400 Subject: [Bro-Dev] script loading changes In-Reply-To: <20110713004021.GD1091@icir.org> References: <3bc0f5e2-79da-4ae9-b7de-c22ea7ca19c2@tangent.ncsa.illinois.edu> <1ffe95af-80c7-4642-8a5d-9a02154e90cb@tangent.ncsa.illinois.edu> <20110713004021.GD1091@icir.org> Message-ID: <4E1D0A54.9000903@ohio.edu> Apologies if I'm missing the point / this already exists :) That being said... >> 3) A new "@add x when y ..." directive that's an alternative to @load, and >> can be used to conditionally load script named 'x' only when the script >> named 'y' has already been loaded. The evaluation of that condition is >> postponed until after all scripts specified via @load or as command line >> arguments have been loaded. 'x' and 'y' can be named according to either >> (1) or (2) above. > Rather than an "@add x when y", what about "@allows x" defined within y, where @allows was evaluated with the stipulations described above? Alternatively, @try x, then define a set of @require directives in x that must be met in order for x to be loaded (but that silently fail / throw a warning if they fail to load)? --Gilbert From hlin33 at illinois.edu Tue Jul 12 20:28:26 2011 From: hlin33 at illinois.edu (Hui Lin) Date: Tue, 12 Jul 2011 20:28:26 -0700 Subject: [Bro-Dev] Hui Lin_Some question on Binpac In-Reply-To: <20110712150954.GB79789@icir.org> References: <20110712150954.GB79789@icir.org> Message-ID: Hi, I do follow the Binpac sample analyzer. And I also notice this part of code which use the value of is_orig to decide whether the packet is the request. When Bro is used to parse network traffic on the network card, it is obvious to see what traffic are coming in and what traffic are going out. But i am using the traffic dump to test, so how can Binpac know which traffic is coming in and going out? I use the Wireshark and actually, wireshark can tell whether the packet is a request or response. Also according to your logic, the Bro considers the originator of the traffic as the client and then this makes sense to my situation. Since the first packet in that dump is a response (without any request before it) and Binpac will consider the source of the packet as the client and the receiver as the server, then this can successfully fool the Binpac to think that this packet is a request. Then the exception can happen. Best, Hui On Tue, Jul 12, 2011 at 8:09 AM, Robin Sommer wrote: > > On Mon, Jul 11, 2011 at 22:10 -0700, you wrote: > > > tell whether the parsing packet is a "request" or a "response". For > example, > > in HTTP protocol, the http header will include such information. > > While that's right, BinPAC/Bro is actually not using that information > to decide which side it's parsing. Generally, Bro always considers the > originator of a connection to be the client side[1], and then it > drives BinPAC parsing accordingly. > > Take a look at http-procotol.pac, there's this piece: > > type HTTP_PDU(is_orig: bool) = case is_orig of { > true -> request: HTTP_Request; > false -> reply: HTTP_Reply; > }; > > And in http-analyzer.pac, there is: > > connection HTTP_Conn(bro_analyzer: BroAnalyzer) { > upflow = HTTP_Flow(true); > downflow = HTTP_Flow(false); > }; > > flow HTTP_Flow(is_orig: bool) { > flowunit = HTTP_PDU(is_orig) withcontext (connection, this); > ... > > } > > > When Bro passes in data, it tells BinPAC which of these flows to use. > From HTTP-binpac.cc: > > void HTTP_Analyzer_binpac::DeliverStream(int len, const u_char* data, > bool orig) > { > .... > // orig is true here if it's originator-side data. > interp->NewData(orig, data, data + len); > } > > There's a skeleton Bro analyzer on the web page that's based on > binpac. If you us that skeleton, you should pretty much automatically > get this direction stuff right. See > http://www.bro-ids.org/development/binpac-sample-analyzer.html > > Robin > > [1] With a few additional heuristics for corner-cases. > > > -- > Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org > ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org > -- Hui Lin Research Assistant DEPEND Research Group, ECE Department University of Illinois at Urbana-Champaign hlin33 at illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20110712/a8fa452c/attachment.html From bro at tracker.bro-ids.org Tue Jul 12 22:19:54 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 13 Jul 2011 05:19:54 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: denneembeNbaw Message-ID: <20110713051952.C6FD036A3A8@taffy.ICSI.Berkeley.EDU> New user registration for user denneembeNbaw -- Bro Tracker Bro Issue Tracker From gregor at icir.org Wed Jul 13 07:45:32 2011 From: gregor at icir.org (Gregor Maier) Date: Wed, 13 Jul 2011 16:45:32 +0200 Subject: [Bro-Dev] script loading changes In-Reply-To: <1ffe95af-80c7-4642-8a5d-9a02154e90cb@tangent.ncsa.illinois.edu> References: <1ffe95af-80c7-4642-8a5d-9a02154e90cb@tangent.ncsa.illinois.edu> Message-ID: <4E1DAF8C.8000304@icir.org> > 1) When an @load or command line argument relies on BROPATH to search for > the script/package, directory separator characters (i.e. '/') must be > replaced with dots ('.'). e.g. "@load frameworks.software". ".bro" > file extensions are still optional. Are dots in directory or script names still allowed (and are they handled correctly?). E.g., what happens if I have a script foo.bar/foo/bar.blub.something.bro And one more question: for debugging: can we have some functionality (e.g., command line argument) that will print all scripts with absolute paths in the order they are loaded? This might be handy in cases were path / script names are ambiguous or a users is wondering which files are loaded exactly. cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From jsiwek at ncsa.illinois.edu Wed Jul 13 08:19:06 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Wed, 13 Jul 2011 10:19:06 -0500 (CDT) Subject: [Bro-Dev] script loading changes In-Reply-To: <20110713004021.GD1091@icir.org> Message-ID: <1442689235.1027.1310570346620.JavaMail.root@zimbra-1.ncsa.uiuc.edu> > > 1) When an @load or command line argument relies on BROPATH to > > search for the script/package, directory separator characters > > (i.e. '/') must be replaced with dots ('.'). > > Do we want to say "may" instead of "must"? So both versions would be > acceptable. So I take this to mean the search order for "@load a.b.c" would be to look in BROPATH for "a.b.c", "a.b.c.bro", "a/b/c", then "a/b/c.bro". And the search order for "@load a/b/c" would be "a/b/c" then "a/b/c.bro" We could do that, but I think in the shipped scripts should use one form consistently -- probably the dotted form or else we'd just be adding a feature that never gets used. And actually I think the search orders listed above should be reversed to make the common case faster. > > 3) A new "@add x when y ..." directive > > One thing we haven't considered yet: what if "x" itself has new @loads > or @adds? Can we make that well-defined? If 'x' contains @loads, they get loaded immediately during the scan of 'x' and set some flag to indicate we should go back to iterating from the beginning of the current pool of @adds-to-evaluate to re-check dependencies. If 'x' contains @adds, then those are added to the end of the pool of @adds-to-evaluate (we should be somewhere in the middle of iterating over the pool at this point). Another problem I was thinking of was what happens when there's @add x when y @add y when z And there's only an "@load z", creating a dependency chain. To resolve those cases, we can take the same approach I describe above and reset the iterator to the beginning of the @adds-to-evaluate pool in order to re-check dependencies whenever one of the elements evaluates successfully. Dependency cycles will just get dropped on the floor if no element on the cycle is ever explicitly @load'd. - Jon From seth at icir.org Wed Jul 13 07:55:26 2011 From: seth at icir.org (Seth Hall) Date: Wed, 13 Jul 2011 10:55:26 -0400 Subject: [Bro-Dev] script loading changes In-Reply-To: <4E1DAF8C.8000304@icir.org> References: <1ffe95af-80c7-4642-8a5d-9a02154e90cb@tangent.ncsa.illinois.edu> <4E1DAF8C.8000304@icir.org> Message-ID: <42DEC64D-EDFE-464E-8B5F-C2727064121A@icir.org> On Jul 13, 2011, at 10:45 AM, Gregor Maier wrote: > And one more question: for debugging: can we have some functionality > (e.g., command line argument) that will print all scripts with absolute > paths in the order they are loaded? This might be handy in cases were > path / script names are ambiguous or a users is wondering which files > are loaded exactly. There is a script named misc/loaded-scripts now that outputs a log file named loaded_scripts.log by default. Determining loaded scripts is all handled in script-land now due to some change Robin did recently. The output still sucks pretty bad, but it's certainly readable. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From jsiwek at ncsa.illinois.edu Wed Jul 13 08:48:33 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Wed, 13 Jul 2011 10:48:33 -0500 (CDT) Subject: [Bro-Dev] script loading changes In-Reply-To: <4E1D0A54.9000903@ohio.edu> Message-ID: <2126581119.1232.1310572113983.JavaMail.root@zimbra-1.ncsa.uiuc.edu> > Apologies if I'm missing the point Oops, I guess I should have described what the point of @add actually was. It's mainly so that it's easier for a script package to define optional components that automatically get loaded when a dependency is met. e.g. the protocols/http package has some scripts that depend on frameworks/software, and those components should be optionally loaded only when frameworks/software has been explicitly loaded instead of a load of protocols/http always resulting in a load of frameworks/software. > Rather than an "@add x when y", what about "@allows x" defined within > y, where @allows was evaluated with the stipulations described above? I think the problem with this is that if a user were developing their own script package that had optional components that depend on e.g. frameworks/software, they wouldn't be able to express that without directly editing something in frameworks/software to @allow their scripts. And that's discouraged. > Alternatively, @try x, then define a set of @require directives in x > that must be met in order for x to be loaded (but that silently fail / > throw a warning if they fail to load)? We want the attempt to load 'x' to be postponed until after the normal loading of @load'd scripts to avoid ordering issues. - Jon From jsiwek at ncsa.illinois.edu Wed Jul 13 09:04:54 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Wed, 13 Jul 2011 11:04:54 -0500 (CDT) Subject: [Bro-Dev] script loading changes In-Reply-To: <4E1DAF8C.8000304@icir.org> Message-ID: <553258985.1411.1310573094190.JavaMail.root@zimbra-1.ncsa.uiuc.edu> > Are dots in directory or script names still allowed (and are they > handled correctly?). E.g., what happens if I have a script > > foo.bar/foo/bar.blub.something.bro Yikes, you're right, that's tough to resolve unless we say dots aren't allowed in package/directory or script names. Or we can escape them in the script: @load foo\.bar.foo.bar\.blub\.something or @load foo\.bar.foo.bar\.blub\.something\.bro I think exposing the dotted format to users this way might be more trouble than it's worth, want to ditch (1) in the original proposal? - Jon From robin at icir.org Wed Jul 13 09:36:23 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 13 Jul 2011 09:36:23 -0700 Subject: [Bro-Dev] script loading changes In-Reply-To: <1442689235.1027.1310570346620.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <20110713004021.GD1091@icir.org> <1442689235.1027.1310570346620.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <20110713163623.GH38860@icir.org> On Wed, Jul 13, 2011 at 10:19 -0500, you wrote: > We could do that, but I think in the shipped scripts should use one > form consistently Yes, definitly; and I'd also use the the dotted form for that. It just seems to make things more consistent (in particular with the relative paths) if we also allow the slashes. > And actually I think the search orders listed above should be reversed > to make the common case faster. Isn't the dotted-case the common one if that's what the default scripts will be using? > If 'x' contains @loads, they get loaded immediately during the scan of 'x' > and set some flag to indicate we should go back to iterating from the > beginning of the current pool of @adds-to-evaluate to re-check dependencies. > If 'x' contains @adds, then those are added to the end of the pool of > @adds-to-evaluate (we should be somewhere in the middle of iterating over > the pool at this point). Sounds good. I can see some quite hard to comprehend dependenices showing up, like globals defined in one script and tests for with @ifdef in another. But I think that's fine, the advantages here weigh more than the opportunity for someone to shoot himself into the foot. :) > And there's only an "@load z", creating a dependency chain. To resolve > those cases, we can take the same approach I describe above and reset Sounds good as well. Let's just make sure to document this all very well. :-) Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Wed Jul 13 09:41:08 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 13 Jul 2011 09:41:08 -0700 Subject: [Bro-Dev] script loading changes In-Reply-To: <553258985.1411.1310573094190.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <4E1DAF8C.8000304@icir.org> <553258985.1411.1310573094190.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <20110713164107.GI38860@icir.org> On Wed, Jul 13, 2011 at 11:04 -0500, you wrote: > Yikes, you're right, that's tough to resolve unless we say dots > aren't allowed in package/directory or script names. What if we always try the literal version first, before trying the alternatives with dots replaced, .bro added, etc? There could still be cases where things get mixed up but it looks unlikley to become an actual problem. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Wed Jul 13 09:47:53 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 13 Jul 2011 09:47:53 -0700 Subject: [Bro-Dev] Hui Lin_Some question on Binpac In-Reply-To: References: <20110712150954.GB79789@icir.org> Message-ID: <20110713164753.GJ38860@icir.org> On Tue, Jul 12, 2011 at 20:28 -0700, you wrote: > to see what traffic are coming in and what traffic are going out. But i am > using the traffic dump to test, so how can Binpac know which traffic is > coming in and going out? Yes, in general that's a problem. Here's what Bro (and thus in turn BinPAC) is doing: (1) by default, the first packet of a new connection indicates the originator. (2) it however also looks at well-known ports: if with (1), the connection would have a well-known port on the originator-side but not on the responder-side, it flips the direction. Is there a well-known port for this protocol? If so, add it to policy/server-ports.bro and it should work. If not, you can trigger the flipping yourslef by calling Connection::FlipRoles(), but you'd need some heuristic to decide who's the client. BinPAC doesn't have anything built in for that (yet). Also, flipping should occur as early as possible, as otherwise things may get mixed up. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Wed Jul 13 10:02:56 2011 From: seth at icir.org (Seth Hall) Date: Wed, 13 Jul 2011 13:02:56 -0400 Subject: [Bro-Dev] Hui Lin_Some question on Binpac In-Reply-To: <20110713164753.GJ38860@icir.org> References: <20110712150954.GB79789@icir.org> <20110713164753.GJ38860@icir.org> Message-ID: On Jul 13, 2011, at 12:47 PM, Robin Sommer wrote: > Is there a well-known port for this protocol? If so, add it to > policy/server-ports.bro and it should work. Eek, good reminder. I need to put that back! .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From jsiwek at ncsa.illinois.edu Wed Jul 13 10:03:14 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Wed, 13 Jul 2011 12:03:14 -0500 (CDT) Subject: [Bro-Dev] script loading changes In-Reply-To: <20110713163623.GH38860@icir.org> Message-ID: <823433455.1961.1310576594039.JavaMail.root@zimbra-1.ncsa.uiuc.edu> > > And actually I think the search orders listed above should be > > reversed to make the common case faster. > > Isn't the dotted-case the common one if that's what the default > scripts will be using? The dotted format is what's used most often in the @load, but usually not what the literal filename/path is going to be. @load frameworks.software.base would do best to look in BROPATH for "frameworks/software/base.bro" first instead of the literal "frameworks.software.base" filename. - Jon From robin at icir.org Wed Jul 13 10:04:37 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 13 Jul 2011 10:04:37 -0700 Subject: [Bro-Dev] script loading changes In-Reply-To: <823433455.1961.1310576594039.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <20110713163623.GH38860@icir.org> <823433455.1961.1310576594039.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <20110713170437.GL38860@icir.org> On Wed, Jul 13, 2011 at 12:03 -0500, you wrote: > The dotted format is what's used most often in the @load, but > usually not what the literal filename/path is going to be. Got it. But actually I don't think it matters much in terms of performance, let's do whatever gives us the best semantics. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From jsiwek at ncsa.illinois.edu Wed Jul 13 10:17:15 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Wed, 13 Jul 2011 12:17:15 -0500 (CDT) Subject: [Bro-Dev] script loading changes In-Reply-To: <20110713164107.GI38860@icir.org> Message-ID: <733511844.2075.1310577435725.JavaMail.root@zimbra-1.ncsa.uiuc.edu> > What if we always try the literal version first, before trying the > alternatives with dots replaced, .bro added, etc? Ok, let me try again. So there's a file in BROPATH with the name foo.bar/foo/bar.blub.something.bro To load that in a script you do @load foo.bar/foo/bar.blub.something And then starting with the literal version it searches BROPATH for 1) foo.bar/foo/bar.blub.something 2) foo.bar/foo/bar.blub.something.bro 3) foo/bar/foo/bar/blub/something 4) foo/bar/foo/bar/blub/something.bro We never make it past #2 because that's the right name. But from a new user's perspective, it's not intuitive to try the literal version because everywhere else, the convention is to reference scripts in packages in the dotted format, which would be: @load foo.bar.foo.bar.blub.something and that would search BROPATH for 1) foo.bar.foo.bar.blub.something 2) foo.bar.foo.bar.blub.something.bro 3) foo/bar/foo/bar/blub/something 4) foo/bar/foo/bar/blub/something.bro And none of those are right. - Jon From robin at icir.org Wed Jul 13 10:48:27 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 13 Jul 2011 10:48:27 -0700 Subject: [Bro-Dev] script loading changes In-Reply-To: <733511844.2075.1310577435725.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <20110713164107.GI38860@icir.org> <733511844.2075.1310577435725.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <20110713174827.GP38860@icir.org> On Wed, Jul 13, 2011 at 12:17 -0500, you wrote: > @load foo.bar/foo/bar.blub.something > 1) foo.bar/foo/bar.blub.something > 2) foo.bar/foo/bar.blub.something.bro > 3) foo/bar/foo/bar/blub/something > 4) foo/bar/foo/bar/blub/something.bro > > We never make it past #2 because that's the right name. Right. > But from a new user's perspective, it's not intuitive to try the > literal version because everywhere else, the convention is to reference > scripts in packages in the dotted format, which would be: > > @load foo.bar.foo.bar.blub.something > > and that would search BROPATH for > > 1) foo.bar.foo.bar.blub.something > 2) foo.bar.foo.bar.blub.something.bro > 3) foo/bar/foo/bar/blub/something > 4) foo/bar/foo/bar/blub/something.bro > > And none of those are right. Yeah, but I at least wouldn't expect anything else than the literal path to work here for foo.bar/foo/bar.blub.something anyway; not sure that's really unintuitive, it's messy to begin with already. :) How about one more tweak to the rules to make this explict: if there's at least one slash in a name, we don't do any dot-to-slash substiations at all. I.e., if you use use actual paths, you need to point to literal files names. If you use dot-style, we'll translate it into paths for you. Does that make sense? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gc355804 at ohio.edu Wed Jul 13 11:11:54 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Wed, 13 Jul 2011 14:11:54 -0400 Subject: [Bro-Dev] script loading changes In-Reply-To: <2126581119.1232.1310572113983.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <2126581119.1232.1310572113983.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <4E1DDFEA.6060507@ohio.edu> On 7/13/2011 11:48 AM, Jonathan Siwek wrote: >> Alternatively, @try x, then define a set of @require directives in x >> that must be met in order for x to be loaded (but that silently fail / >> throw a warning if they fail to load)? > We want the attempt to load 'x' to be postponed until after the normal > loading of @load'd scripts to avoid ordering issues. > Yeah; was thinking that @require could evaluate after all @load's have been completed, very similar to @load x when y. More relevant, though: I'm kind of worried that @load x when y would break logical encapsulation a little bit (since it seems like 'load x when y' essentially means that something other than x needs to be aware of exactly what x needs to load in order to run). I guess that's a matter of preference, though. --Gilbert From jsiwek at ncsa.illinois.edu Wed Jul 13 11:20:32 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Wed, 13 Jul 2011 13:20:32 -0500 (CDT) Subject: [Bro-Dev] script loading changes In-Reply-To: <20110713174827.GP38860@icir.org> Message-ID: <485044793.2442.1310581232579.JavaMail.root@zimbra-1.ncsa.uiuc.edu> > Yeah, but I at least wouldn't expect anything else than the literal > path to work here for foo.bar/foo/bar.blub.something anyway; not sure > that's really unintuitive, it's messy to begin with already. :) Yeah, good point. I think e.g. python doesn't even really handle crazy stuff like that well, so the answer of "don't try naming packages and scripts with dots" is probably ok. > How about one more tweak to the rules to make this explict: if there's > at least one slash in a name, we don't do any dot-to-slash > substiations at all. I.e., if you use use actual paths, you need to > point to literal files names. If you use dot-style, we'll translate it > into paths for you. Does that make sense? Yes, that seems right to me. - Jon From vern at icir.org Wed Jul 13 11:27:20 2011 From: vern at icir.org (Vern Paxson) Date: Wed, 13 Jul 2011 11:27:20 -0700 Subject: [Bro-Dev] .'s vs. /'s (Re: script loading changes) In-Reply-To: <823433455.1961.1310576594039.JavaMail.root@zimbra-1.ncsa.uiuc.edu> (Wed, 13 Jul 2011 12:03:14 CDT). Message-ID: <20110713182720.BA27236A3A8@taffy.ICSI.Berkeley.EDU> Dipping into this thread (so maybe missing important context - apologies if so), this is seeming pretty weird: > @load frameworks.software.base > > would do best to look in BROPATH for "frameworks/software/base.bro" > first instead of the literal "frameworks.software.base" filename. Why isn't the answer that this should just be @load frameworks/software/base ? What's so cool about using '.'s instead of '/'s that it's worth conflicting with the user's file-system mental model? Vern From bro at tracker.bro-ids.org Wed Jul 13 12:44:17 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 13 Jul 2011 19:44:17 -0000 Subject: [Bro-Dev] #493: RemoteSerializer::Log method should give peer to remote_log event Message-ID: <046.92af99e6494fd1fafea993dd0397bf7b@tracker.bro-ids.org> #493: RemoteSerializer::Log method should give peer to remote_log event -----------------------------+-------------------- Reporter: seth | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: | -----------------------------+-------------------- Right now the RemoteSerializer::Log method adds some peer information to the log message directly. This method should not be adding this data directly and instead pass along a peer record to the remote_log event. -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Wed Jul 13 14:11:39 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Wed, 13 Jul 2011 16:11:39 -0500 (CDT) Subject: [Bro-Dev] script loading changes In-Reply-To: <4E1DDFEA.6060507@ohio.edu> Message-ID: <356289713.3411.1310591499719.JavaMail.root@zimbra-1.ncsa.uiuc.edu> > More relevant, though: I'm kind of worried that @load x when y would > break logical encapsulation a little bit (since it seems like 'load x > when y' essentially means that something other than x needs to be > aware of exactly what x needs to load in order to run). "@add x when y" isn't actually saying that y is required for x to work (I guess generally it will be), but rather that there's a condition that must be met before loading 'x'. But I think you're right that, for how we'd use it, it's weird for 'x' itself to not be maintaining the conditions that are required to load it. It's going to be hard to maintain "@add x when y" statements because you have to look whether you need to change any whenever you change what 'x' is loading. Let's step back for a second, I'm not seeing why we're coming up with such complex solutions anymore. The problem we're trying to solve is how to best organize the packages under protocol/ such that there's a way to load them without a possibly unwanted side-effect of loading an entire package under frameworks/ because one script happens to require it (is that right Seth?). e.g. we want "@load protocols/http" to avoid loading frameworks/software. Why don't we just write the __load__.bro manifest of the package to not @load scripts that may have unwanted side effects? The __load__.bro that gets loaded implicitly is already provided only for convenience and may not actually load the entire contents of the package. I think a user should have three choices: 1) pick & choose exactly what scripts you want from a package 2) rely on a package's __load__.bro to pick a minimal set of scripts from the package that best reflect its core functionality e.g. via "@load protocols/http" 3) the "I don't care how much extra stuff gets loaded, give me everything in the package" approach via "@load protocols/http/*" to recursively load all files that have a .bro extension (skipping __load__.bro) Only #3 is new functionality. These choices make it quite clear to the user what to expect, while the @add/when conditional loading stuff is going to appear a bit magical and hard to troubleshoot. - Jon From jsiwek at ncsa.illinois.edu Wed Jul 13 14:20:32 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Wed, 13 Jul 2011 16:20:32 -0500 (CDT) Subject: [Bro-Dev] .'s vs. /'s (Re: script loading changes) In-Reply-To: <20110713182720.BA27236A3A8@taffy.ICSI.Berkeley.EDU> Message-ID: <446815139.3473.1310592032777.JavaMail.root@zimbra-1.ncsa.uiuc.edu> > ? What's so cool about using '.'s instead of '/'s that it's worth > conflicting with the user's file-system mental model? Familiarity with other languages (e.g. python modules) is the main reason. I'm on the fence as to whether the benefit of that is worth it. - Jon From gc355804 at ohio.edu Wed Jul 13 17:28:12 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Wed, 13 Jul 2011 20:28:12 -0400 Subject: [Bro-Dev] .'s vs. /'s (Re: script loading changes) In-Reply-To: <446815139.3473.1310592032777.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <446815139.3473.1310592032777.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <4E1E381C.3070901@ohio.edu> On 7/13/2011 5:20 PM, Jonathan Siwek wrote: >> ? What's so cool about using '.'s instead of '/'s that it's worth >> conflicting with the user's file-system mental model? I'm honestly not too sure how the new scripting stuff works, exactly, so forgive my ignorance. I'm also not a language design guy. That being said, I disagree that the file-system mental model should be applied to a scripting language that uses packages / modules. The idea is that a package is meant to abstract away certain path details that are inherent to things like #include; instead of worrying about files, you worry about code constructs. For example, bro.http.headers may not exist as a file on its own; instead, bro.http.headers may exist in "./bro/http/extensions.bro" As such, I see a huge difference between @load a/bro/module and @load a.bro.module: one loads a file, and the other loads a construct. As it happens, if code is written well, there's a lot of overlap there. . . but there's still a big conceptual difference. I think I'd tentatively argue that, if '.' and '/' are completely interchangeable (and *must* remain that way), bro scripts haven't yet gotten to the point where they should worry about using '.' to load stuff. If anything, I think folks trying to learn bro script with python / java / etc. experience could find the syntax deceptive: "I don't think that . means what you think it means." Just my $0.02, though. --Gilbert From seth at icir.org Wed Jul 13 18:12:09 2011 From: seth at icir.org (Seth Hall) Date: Wed, 13 Jul 2011 21:12:09 -0400 Subject: [Bro-Dev] script loading changes In-Reply-To: <356289713.3411.1310591499719.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <356289713.3411.1310591499719.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <5FCA386C-1162-4826-9EEF-BC882045B3E6@icir.org> On Jul 13, 2011, at 5:11 PM, Jonathan Siwek wrote: > The problem we're trying to solve is how to best organize the packages > under protocol/ such that there's a way to load them without a possibly > unwanted side-effect of loading an entire package under frameworks/ > because one script happens to require it (is that right Seth?). Correct. > e.g. we want "@load protocols/http" to avoid loading frameworks/software. > > Why don't we just write the __load__.bro manifest of the package to not > @load scripts that may have unwanted side effects? The __load__.bro that > gets loaded implicitly is already provided only for convenience and may > not actually load the entire contents of the package. The __load__.bro scripts are actually where I would be hiding these extra dependency based loads. I was just looking for a way to reduce dependencies without having to add code to manage things manually. For example, I'd like packages to offer the ability to do something with the software framework if it's available, but if it's not available to just not load the functionality. I'm starting to recognize that I'm looking to do something like dependency injection but perhaps we're still a bit early on in the modularization of Bro scripts for us to figure out how to do this well. If everyone's onboard we can just table this discussion for now and I can do this more manually, I may even be able to rig up a dependency injection-like framework in script-land. I would still really like the relative path based loading though, it's really painful to do inter-directory script loading right now and it would be a bit difficult for others to contribute external scripts and manage them well because the full path is used for loading. > 3) the "I don't care how much extra stuff gets loaded, give me everything > in the package" approach via "@load protocols/http/*" to recursively > load all files that have a .bro extension (skipping __load__.bro) I'm not sure this is a good option as I have written a couple of things where you would never want to load everything and loading everything could actually lead to disaster (or at least broken-ness). :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From jsiwek at ncsa.illinois.edu Wed Jul 13 19:37:54 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Wed, 13 Jul 2011 21:37:54 -0500 (CDT) Subject: [Bro-Dev] script loading changes In-Reply-To: <5FCA386C-1162-4826-9EEF-BC882045B3E6@icir.org> Message-ID: <705749737.4081.1310611074819.JavaMail.root@zimbra-1.ncsa.uiuc.edu> > example, I'd like packages to offer the ability to do something with > the software framework if it's available, but if it's not available to > just not load the functionality. It seems like that approach is well-intended at making packages appear more modular, but weren't the solutions we came up with taking away from a user's ability to control what scripts get loaded? For example, what if a user wants the ssh package's software-related scripts to load, but not the http package's software-related scripts? > I'm starting to recognize that I'm looking to do something like > dependency injection Dependency injection patterns are more for object-oriented languages, no? > but perhaps we're still a bit early on in the > modularization of Bro scripts for us to figure out how to do this > well. If everyone's onboard we can just table this discussion for now Yeah, we might wait. I'm not fully convinced that the way you currently organized things actually has a modularity problem that significantly hinders a user's ability to use bro. > I would still really like the relative path based loading though Yeah, I didn't have any problems with that new feature. - Jon From seth at icir.org Wed Jul 13 20:14:20 2011 From: seth at icir.org (Seth Hall) Date: Wed, 13 Jul 2011 23:14:20 -0400 Subject: [Bro-Dev] script loading changes In-Reply-To: <705749737.4081.1310611074819.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <705749737.4081.1310611074819.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: On Jul 13, 2011, at 10:37 PM, Jonathan Siwek wrote: > It seems like that approach is well-intended at making packages appear > more modular, but weren't the solutions we came up with taking away > from a user's ability to control what scripts get loaded? > For example, what if a user wants the ssh package's software-related > scripts to load, but not the http package's software-related scripts? Yep, agreed on both counts. I had kind of identified the "globally enabled functionality" problem myself internally, but was reluctant to admit it to myself because in the case of the software scripts it should cause very little overhead. It does cry out for a more generic solution though that leaves control in the users hands. > Dependency injection patterns are more for object-oriented languages, no? Yeah, I think you're right. :) >> but perhaps we're still a bit early on in the >> modularization of Bro scripts for us to figure out how to do this >> well. If everyone's onboard we can just table this discussion for now > > Yeah, we might wait. I'm not fully convinced that the way you currently > organized things actually has a modularity problem that significantly > hinders a user's ability to use bro. At least we're slowly getting somewhere and sometimes by choosing *not* to do things. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Wed Jul 13 21:02:03 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 14 Jul 2011 04:02:03 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: AnimeMan Message-ID: <20110714040201.4489236A379@taffy.ICSI.Berkeley.EDU> New user registration for user AnimeMan -- Bro Tracker Bro Issue Tracker From robin at icir.org Wed Jul 13 21:45:26 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 13 Jul 2011 21:45:26 -0700 Subject: [Bro-Dev] script loading changes In-Reply-To: References: <705749737.4081.1310611074819.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <20110714044526.GB61438@icir.org> On Wed, Jul 13, 2011 at 23:14 -0400, you wrote: > At least we're slowly getting somewhere and sometimes by choosing *not* to do things. :) Sorry for being the party pooper now but I *do* like the ability to load something automatically (only) when something else is loaded as well. I don't think that can really be substituted for at the scripting layer right now, at least not nicely. software.bro is a good application for that and I'm sure there are more (like the intelligence framework?). We need to bring the current prefix mechanism back to live anyway (and with the new load order I think), and this would be quite easy to add on top of that if I'm not missing anything. So why don't we give it a try and see if it works for what's it's supposed to help with. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Wed Jul 13 21:53:55 2011 From: seth at icir.org (Seth Hall) Date: Thu, 14 Jul 2011 00:53:55 -0400 Subject: [Bro-Dev] script loading changes In-Reply-To: <20110714044526.GB61438@icir.org> References: <705749737.4081.1310611074819.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110714044526.GB61438@icir.org> Message-ID: <22CBAC4F-7F3C-4A1E-A3DE-B0ACCBE54238@icir.org> On Jul 14, 2011, at 12:45 AM, Robin Sommer wrote: > Sorry for being the party pooper now but I *do* like the ability to > load something automatically (only) when something else is loaded as > well. Haha, I don't think it's possible for you to be the Bro party pooper. You're the life of the party! > I don't think that can really be substituted for at the > scripting layer right now, at least not nicely. I'll think about this. It may be possible to make something fairly clean at the scripting layer. > software.bro is a good > application for that and I'm sure there are more (like the > intelligence framework?). Yep, the intelligence framework is another big one. The metrics framework may be another one as well, but possibly not. I'm not sure of how exactly it will be used yet. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From robin at icir.org Wed Jul 13 21:58:19 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 13 Jul 2011 21:58:19 -0700 Subject: [Bro-Dev] script loading changes In-Reply-To: <22CBAC4F-7F3C-4A1E-A3DE-B0ACCBE54238@icir.org> References: <705749737.4081.1310611074819.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110714044526.GB61438@icir.org> <22CBAC4F-7F3C-4A1E-A3DE-B0ACCBE54238@icir.org> Message-ID: <20110714045819.GE61438@icir.org> On Thu, Jul 14, 2011 at 00:53 -0400, you wrote: > I'll think about this. It may be possible to make something fairly > clean at the scripting layer. Ok, if you can come up with something nice, I'm all fine. :-) Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From jsiwek at ncsa.illinois.edu Thu Jul 14 08:03:12 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Thu, 14 Jul 2011 10:03:12 -0500 (CDT) Subject: [Bro-Dev] script loading changes In-Reply-To: <20110714044526.GB61438@icir.org> Message-ID: <88db96a1-e0dd-4613-aed7-6682697c3f23@tangent.ncsa.illinois.edu> > We need to bring the current prefix mechanism back to live anyway > (and with the new load order I think), and this would be quite easy to add > on top of that if I'm not missing anything. In my mind I saw the new implementation of @prefixes as independent of that "@add x when y" idea. There's some similarities in how they work, but they don't depend on each other. > So why don't we give it a try For now I'll attempt these changes: - allow relative paths in @load's, e.g. "@load ./software" - new @prefixes implementation I'll put these off due to contention: - allowing '.' vs. '/' in package names - "@add x when y" From seth at icir.org Thu Jul 14 08:14:43 2011 From: seth at icir.org (Seth Hall) Date: Thu, 14 Jul 2011 11:14:43 -0400 Subject: [Bro-Dev] script loading changes In-Reply-To: <88db96a1-e0dd-4613-aed7-6682697c3f23@tangent.ncsa.illinois.edu> References: <88db96a1-e0dd-4613-aed7-6682697c3f23@tangent.ncsa.illinois.edu> Message-ID: <65DF236D-BB2B-42A5-BAB8-CA29295FFE63@icir.org> On Jul 14, 2011, at 11:03 AM, Jonathan Siwek wrote: > For now I'll attempt these changes: > > - allow relative paths in @load's, e.g. "@load ./software" > - new @prefixes implementation > > I'll put these off due to contention: > > - allowing '.' vs. '/' in package names > - "@add x when y" That sounds good to me. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From robin at icir.org Thu Jul 14 08:41:53 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 14 Jul 2011 08:41:53 -0700 Subject: [Bro-Dev] script loading changes In-Reply-To: <88db96a1-e0dd-4613-aed7-6682697c3f23@tangent.ncsa.illinois.edu> References: <20110714044526.GB61438@icir.org> <88db96a1-e0dd-4613-aed7-6682697c3f23@tangent.ncsa.illinois.edu> Message-ID: <20110714154153.GD78024@icir.org> On Thu, Jul 14, 2011 at 10:03 -0500, you wrote: > "@add x when y" idea. There's some similarities in how they work, but they > don't depend on each other. That's right, they don't depend on each other. I was just thinking that doing the @add doesn't require much in addition. > For now I'll attempt these changes: Ok. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Thu Jul 14 10:34:03 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 14 Jul 2011 10:34:03 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/script-load-changes: Make @load statements recognize relative paths. (e5e3bf2) In-Reply-To: <201107141641.p6EGfNNR009344@bro-ids.icir.org> References: <201107141641.p6EGfNNR009344@bro-ids.icir.org> Message-ID: <20110714173403.GP78024@icir.org> On Thu, Jul 14, 2011 at 09:41 -0700, Jonathan Siwek wrote: > +# @TEST-EXEC: echo "@load ./bar" > foo/foo.bro > +# @TEST-EXEC: echo "@load ../baz" >> foo/foo.bro Jfyi in case you haven't seen it yet: btest has a command "@TEST-START-FILE " to create additional files for the test, see http://www.bro-ids.org/development/btest.html#writing-tests Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Thu Jul 14 20:09:03 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 15 Jul 2011 03:09:03 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: edunteete Message-ID: <20110715030902.034FB369FC9@taffy.ICSI.Berkeley.EDU> New user registration for user edunteete -- Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Jul 15 04:09:28 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 15 Jul 2011 11:09:28 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: deropilaw Message-ID: <20110715110928.BCD3036A390@taffy.ICSI.Berkeley.EDU> New user registration for user deropilaw -- Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Jul 15 10:44:17 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 15 Jul 2011 17:44:17 -0000 Subject: [Bro-Dev] #475: Update Broccoli Test Scripts In-Reply-To: <048.a284abbfb80e3c325b0a34424a7f512f@tracker.bro-ids.org> References: <048.a284abbfb80e3c325b0a34424a7f512f@tracker.bro-ids.org> Message-ID: <063.fcb1ad417a7769d9ac2ee0e7598da4f4@tracker.bro-ids.org> #475: Update Broccoli Test Scripts ----------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Broccoli | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by jsiwek): * priority: Normal => Low * version: => git/master * type: Task => Merge Request Comment: These are updated on branch: `topic/jsiwek/istate-tests-update` in repositories: `bro`, `broccoli`, `broccoli-python` -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Jul 17 05:09:21 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 17 Jul 2011 12:09:21 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: loobeGobtredo Message-ID: <20110717120921.9851D36A380@taffy.ICSI.Berkeley.EDU> New user registration for user loobeGobtredo -- Bro Tracker Bro Issue Tracker From matthias at vallentin.net Sun Jul 17 08:32:43 2011 From: matthias at vallentin.net (Matthias Vallentin) Date: Sun, 17 Jul 2011 08:32:43 -0700 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <20110708132228.GA84600@icir.org> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> <20110707215133.GF77684@icir.org> <4E163B71.6050804@ohio.edu> <20110708132228.GA84600@icir.org> Message-ID: <1310914521-sup-1714@samurai.icir.org> Although it seems that folks have settled on pthreads, I still would like to add my two cents, biased by my own experience with C++11 (aka C++0x). On Jul 08 2011 at 06:22AM PDT, Robin Sommer wrote: > I forgot the obvious one yesterday: Intel's TBB. That's what the > multi-core Bro prototype is already using, and it's main thread > abstraction is (almost?) compatible to C++0x. I've been using C++11 for quite a while now and can only say that it feels like it was overdue. One writes much less boilerplate for what one wants to achieve (e.g., functors) and many mature components from the Boost libraries found their way into the C++11 standard library: threads, tasks/futures, smart pointers, SFINAE helpers like boost::enable_if, RNGs, etc. Overall, I find myself needing less time to write more code that actually does something. Except for the thread-safe data structures (which we can wrap ourselves, e.g., thread-safe queues) and the TBB scheduler (which we don't plan to use IIRC), C++11 meets our needs from a feature standpoint. The benefit would be avoiding licensing hassles and reaping the, erm, somewhat underappreciated improvements that come for free :-). Now that the standard is sealed, everybody will use C++11 in a few years and widely used compilers like gcc will have implemented the full standard. I just want to point out that we might oversee a free lunch down the road. > But somebody has maintain the code that's *using* Boost ... My main > concern is actually that once we have Boost, folks will immediately > start using pretty much any feature it provides. :-) The main problem I see here is that Boost is a mixed bag: some libraries are really high quality (and thus made their way into the new standard) while others still need time to mature and are of experimental nature. If we really want to use Boost (say to have a platform-independent networking library), we could whitelist the Boost components that we allow in Bro. While we're at it, Boost Asio is a nice library not only for platform-independent networking, but also to structure computation at the granularity of tasks. Unlike 0mq, it is not a messaging subsystem, we would need to obtain this functionality elsewhere. I have not looked at it in detail, but 0mq is just one of many options in the space of messaging middle layers [1]. Matthias [1] http://stackoverflow.com/questions/731233/activemq-or-rabbitmq-or-zeromq-or From bro at tracker.bro-ids.org Sun Jul 17 22:53:24 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 18 Jul 2011 05:53:24 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: konowilevsgoa Message-ID: <20110718055322.EA6BA36A38E@taffy.ICSI.Berkeley.EDU> New user registration for user konowilevsgoa -- Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Jul 17 23:33:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 18 Jul 2011 06:33:13 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: Bypesophish Message-ID: <20110718063311.34AE636A394@taffy.ICSI.Berkeley.EDU> New user registration for user Bypesophish -- Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Jul 18 00:02:09 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 18 Jul 2011 07:02:09 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: Endabylal Message-ID: <20110718070207.21EC736A39D@taffy.ICSI.Berkeley.EDU> New user registration for user Endabylal -- Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Jul 18 06:55:12 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 18 Jul 2011 13:55:12 -0000 Subject: [Bro-Dev] #494: topic/remove-tabs - Tabs in broccoli-python may have caused trouble. Message-ID: <046.8ac9b3e0460e4ca13a0af895fb855b81@tracker.bro-ids.org> #494: topic/remove-tabs - Tabs in broccoli-python may have caused trouble. -----------------------------+------------------------ Reporter: seth | Owner: robin Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: broccoli-python | Version: git/master Keywords: | -----------------------------+------------------------ I think there was some strange stuff being caused by accidental tabs in the broccoli.py script so I replaced them with spaces. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Jul 18 07:51:18 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 18 Jul 2011 14:51:18 -0000 Subject: [Bro-Dev] #451: Remove DNS options for skipping auth/addl events In-Reply-To: <047.8ae817995be2255322e68a394e45df3e@tracker.bro-ids.org> References: <047.8ae817995be2255322e68a394e45df3e@tracker.bro-ids.org> Message-ID: <062.6c1c38560598c5d8d2fa478ee310ce0c@tracker.bro-ids.org> #451: Remove DNS options for skipping auth/addl events ---------------------+-------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: Resolution: | Keywords: ---------------------+-------------------- Changes (by seth): * milestone: Bro1.6 => Bro1.7 Comment: low priority, bumping it back. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Jul 18 08:18:30 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 18 Jul 2011 15:18:30 -0000 Subject: [Bro-Dev] #488: Bug in HTTP parser In-Reply-To: <047.adda44ca13a0bcadd2291eced501c487@tracker.bro-ids.org> References: <047.adda44ca13a0bcadd2291eced501c487@tracker.bro-ids.org> Message-ID: <062.3384d8f71f65741f7413a3211ebab417@tracker.bro-ids.org> #488: Bug in HTTP parser ----------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by seth): Did this get committed? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Jul 18 08:20:21 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 18 Jul 2011 15:20:21 -0000 Subject: [Bro-Dev] #474: &raw_output turns null values into \0 In-Reply-To: <046.21443edb25709a7dff08ac2dadf6c6e0@tracker.bro-ids.org> References: <046.21443edb25709a7dff08ac2dadf6c6e0@tracker.bro-ids.org> Message-ID: <061.0320cdc931af7158715ad961a438079d@tracker.bro-ids.org> #474: &raw_output turns null values into \0 ----------------------+--------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: preview ----------------------+--------------------- Changes (by seth): * keywords: => preview Comment: This bug could lead to the entity extraction scripts failing in some cases. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Jul 18 08:22:25 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 18 Jul 2011 15:22:25 -0000 Subject: [Bro-Dev] #486: Prefixes don't support directories In-Reply-To: <047.e78f97399b2dd07d57a5512021972454@tracker.bro-ids.org> References: <047.e78f97399b2dd07d57a5512021972454@tracker.bro-ids.org> Message-ID: <062.4c366254d33c0210ded4906934a091e8@tracker.bro-ids.org> #486: Prefixes don't support directories ----------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: preview ----------------------+------------------------ Changes (by seth): * keywords: => preview Comment: This implementation just needs merged prior to the preview. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Jul 18 08:23:21 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 18 Jul 2011 15:23:21 -0000 Subject: [Bro-Dev] #465: Fix up the MIME analyzer In-Reply-To: <046.cab68644f8f2eb3e5bb44bf373b6902d@tracker.bro-ids.org> References: <046.cab68644f8f2eb3e5bb44bf373b6902d@tracker.bro-ids.org> Message-ID: <061.6dea866853a3e926b1df6b4e31289027@tracker.bro-ids.org> #465: Fix up the MIME analyzer ----------------------+------------------------ Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Changes (by seth): * milestone: Bro1.6 => Bro1.7 Comment: This is mostly an internal detail, let's push it back to the next release. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Jul 18 08:25:24 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 18 Jul 2011 15:25:24 -0000 Subject: [Bro-Dev] #489: InternalWarning causing most shutdowns In-Reply-To: <046.707fba065aa505bef5d93d6693f2e58f@tracker.bro-ids.org> References: <046.707fba065aa505bef5d93d6693f2e58f@tracker.bro-ids.org> Message-ID: <061.1fa0904a474cdaad5e96c3b6dd4f1d29@tracker.bro-ids.org> #489: InternalWarning causing most shutdowns -----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Rejected | Keywords: -----------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Rejected Comment: Ugh, I think this was the bug that I was causing. I haven't seen anything crashes due to it. I'l going to close this ticket. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Jul 18 08:27:21 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 18 Jul 2011 15:27:21 -0000 Subject: [Bro-Dev] #491: Another logging framework segfault In-Reply-To: <046.62907b162b4bdc21b0d64b22b008e9be@tracker.bro-ids.org> References: <046.62907b162b4bdc21b0d64b22b008e9be@tracker.bro-ids.org> Message-ID: <061.bfbc4b3d2a664e9516584d4287cd63ac@tracker.bro-ids.org> #491: Another logging framework segfault -----------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: I believe this was fixed in commit f83650f14ad18093598b2f69c17a1b20b044a10c -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Jul 18 08:28:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 18 Jul 2011 15:28:13 -0000 Subject: [Bro-Dev] #472: Cleanup scripts in bro-aux In-Reply-To: <047.3f213fc8d7000dd46ed9e7cf4dcc81a0@tracker.bro-ids.org> References: <047.3f213fc8d7000dd46ed9e7cf4dcc81a0@tracker.bro-ids.org> Message-ID: <062.2807423d418d68968a5fd870ef590ef8@tracker.bro-ids.org> #472: Cleanup scripts in bro-aux ---------------------+--------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: preview ---------------------+--------------------- Changes (by seth): * keywords: => preview -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Jul 18 10:36:15 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 18 Jul 2011 17:36:15 -0000 Subject: [Bro-Dev] #472: Cleanup scripts in bro-aux In-Reply-To: <047.3f213fc8d7000dd46ed9e7cf4dcc81a0@tracker.bro-ids.org> References: <047.3f213fc8d7000dd46ed9e7cf4dcc81a0@tracker.bro-ids.org> Message-ID: <062.d2a7b2669e93fe4fc8dafb5fad9ff9e5@tracker.bro-ids.org> #472: Cleanup scripts in bro-aux ---------------------+--------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: preview ---------------------+--------------------- Comment (by jsiwek): By "cleanup" does that mean some need to be updated to work or that some can be removed because they're no longer necessary? To me, the later seems to be common: * bro_logchk: not sure this is still helpful with the new HTTP/FTP log formats -- grep/awk could probably easily replace a lot of what it does * host-grep: I think this just doesn't work because new logs are tab delimited and the final `grep -E` anchors the regex with spaces. I think the -a option is broken irrespective of new policy script log format. I can submit a fix for both, but I'd need clarification on what the -a option is really supposed to do. * host-to-addrs: still fine * hot-report: is this still applicable? It also depends on `hf` and `cf`, so at least it probably doesn't make sense to maintain it in this repo anymore since those utils aren't either? * ip-grep: still fine * mon-report: not sure if this is still needed, but seems like the only reason why it wouldn't work is that it depends on either `host-grep` or `hot-report` * lock_file: seems fine * mvlog: can't this now be done entirely with the new logging framework's postprocessor hooks? Anything else I overlooked? -- Ticket URL: Bro Tracker Bro Issue Tracker From vern at icir.org Mon Jul 18 10:36:59 2011 From: vern at icir.org (Vern Paxson) Date: Mon, 18 Jul 2011 10:36:59 -0700 Subject: [Bro-Dev] search functionality for web page Message-ID: <20110718173659.B0FC536A3A7@taffy.ICSI.Berkeley.EDU> It would be nice if the home page included a search box. Is this something we can provide? Vern From vallentin at icir.org Mon Jul 18 11:05:19 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Mon, 18 Jul 2011 11:05:19 -0700 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <20110708132228.GA84600@icir.org> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> <20110707215133.GF77684@icir.org> <4E163B71.6050804@ohio.edu> <20110708132228.GA84600@icir.org> Message-ID: <1311012198-sup-8316@samurai.icir.org> Although it seems that folks have settled on pthreads, I still would like to add my two cents, biased by my own experience with C++11 (aka C++0x). On Jul 08 2011 at 06:22AM PDT, Robin Sommer wrote: > I forgot the obvious one yesterday: Intel's TBB. That's what the > multi-core Bro prototype is already using, and it's main thread > abstraction is (almost?) compatible to C++0x. I've been using C++11 for quite a while now and can only say that it feels like it was overdue. One writes much less boilerplate for what one wants to achieve (e.g., functors) and many mature components from the Boost libraries found their way into the C++11 standard library: threads, tasks/futures, smart pointers, SFINAE helpers like boost::enable_if, RNGs, etc. Overall, I find myself needing less time to write more code that actually does something. Except for the thread-safe data structures (which we can wrap ourselves, e.g., thread-safe queues) and the TBB scheduler (which we don't plan to use IIRC), C++11 meets our needs from a feature standpoint. The benefit would be avoiding licensing hassles and reaping the, erm, somewhat underappreciated improvements that come for free :-). Now that the standard is sealed, everybody will use C++11 in a few years and widely used compilers like gcc will have implemented the full standard. I just want to point out that we might oversee a free lunch down the road. > But somebody has maintain the code that's *using* Boost ... My main > concern is actually that once we have Boost, folks will immediately > start using pretty much any feature it provides. :-) The main problem I see here is that Boost is a mixed bag: some libraries are really high quality (and thus made their way into the new standard) while others still need time to mature and are of experimental nature. If we really want to use Boost (say to have a platform-independent networking library), we could whitelist the Boost components that we allow in Bro. While we're at it, Boost Asio is a nice library not only for platform-independent networking, but also to structure computation at the granularity of tasks. It also plays nicely with C++11, i.e., facilities the implementation of user-space thread scheduling. Unlike 0mq, it does not feature a messaging subsystem; we would need to obtain this functionality elsewhere. Matthias From robin at icir.org Mon Jul 18 11:38:05 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 18 Jul 2011 11:38:05 -0700 Subject: [Bro-Dev] search functionality for web page In-Reply-To: <20110718173659.B0FC536A3A7@taffy.ICSI.Berkeley.EDU> References: <20110718173659.B0FC536A3A7@taffy.ICSI.Berkeley.EDU> Message-ID: <20110718183805.GN40762@icir.org> On Mon, Jul 18, 2011 at 10:36 -0700, you wrote: > It would be nice if the home page included a search box. Is this something > we can provide? If it's fine to just point to Google with a "+site:bro-ids.org" query, then I think yes. I don't really want to set up our own search system though (because of work it takes and risk of getting something wrong). Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Mon Jul 18 11:50:10 2011 From: seth at icir.org (Seth Hall) Date: Mon, 18 Jul 2011 14:50:10 -0400 Subject: [Bro-Dev] search functionality for web page In-Reply-To: <20110718183805.GN40762@icir.org> References: <20110718173659.B0FC536A3A7@taffy.ICSI.Berkeley.EDU> <20110718183805.GN40762@icir.org> Message-ID: <627D4AD7-FB11-427A-8266-65434B77EB6A@icir.org> On Jul 18, 2011, at 2:38 PM, Robin Sommer wrote: > If it's fine to just point to Google with a "+site:bro-ids.org" query, > then I think yes. I don't really want to set up our own search system > though (because of work it takes and risk of getting something wrong). We should probably use the google custom search engine thing. We can make it look like our site apparently, but it's easy to make it search a large set of sites. Here's the list of URLs that I think we should have a search include... http://twitter.com/#!/Bro_IDS http://blog.bro-ids.org/ http://twitter.com/#!/search/%23BroIDS http://mailman.icsi.berkeley.edu/mailman/listinfo/bro http://git.bro-ids.org http://tracker.bro-ids.org http://www.bro-ids.org Any others? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Mon Jul 18 13:18:48 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 18 Jul 2011 20:18:48 -0000 Subject: [Bro-Dev] #255: internal error: NB-DNS error in DNS_Mgr::Process (recvfrom(): Connection refused) In-Reply-To: <056.417dfe2f09777f9f1b0a836ed664bcc7@tracker.bro-ids.org> References: <056.417dfe2f09777f9f1b0a836ed664bcc7@tracker.bro-ids.org> Message-ID: <071.5633839239296ef5fd3fb8304862eed9@tracker.bro-ids.org> #255: internal error: NB-DNS error in DNS_Mgr::Process (recvfrom(): Connection refused) -----------------------------+---------------------- Reporter: Tyler.Schoenke | Owner: robin Type: Problem | Status: accepted Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: 1.5.2 Resolution: | Keywords: -----------------------------+---------------------- Changes (by robin): * owner: => robin * status: new => accepted -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Jul 18 13:20:00 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 18 Jul 2011 20:20:00 -0000 Subject: [Bro-Dev] #469: Core IRC analyzer needs to provide is_orig field. In-Reply-To: <046.ead7a0f871260990c028a605e157ef4b@tracker.bro-ids.org> References: <046.ead7a0f871260990c028a605e157ef4b@tracker.bro-ids.org> Message-ID: <061.3bda1521b1d86b6bb2525c7f3eb75474@tracker.bro-ids.org> #469: Core IRC analyzer needs to provide is_orig field. ----------------------+------------------------ Reporter: seth | Owner: jsiwek Type: Problem | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: preview ----------------------+------------------------ Changes (by seth): * keywords: => preview * owner: => jsiwek * status: new => assigned -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Mon Jul 18 13:57:32 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 18 Jul 2011 13:57:32 -0700 Subject: [Bro-Dev] search functionality for web page In-Reply-To: <627D4AD7-FB11-427A-8266-65434B77EB6A@icir.org> References: <20110718173659.B0FC536A3A7@taffy.ICSI.Berkeley.EDU> <20110718183805.GN40762@icir.org> <627D4AD7-FB11-427A-8266-65434B77EB6A@icir.org> Message-ID: <20110718205732.GC48547@icir.org> On Mon, Jul 18, 2011 at 14:50 -0400, you wrote: > We should probably use the google custom search engine thing. Good idea, I'll look into that. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From vern at icir.org Tue Jul 19 09:28:26 2011 From: vern at icir.org (Vern Paxson) Date: Tue, 19 Jul 2011 09:28:26 -0700 Subject: [Bro-Dev] build problem on Snow Leopard Message-ID: <20110719162826.1364036A37D@taffy.ICSI.Berkeley.EDU> On a basically fresh checkout (I had grabbed it originally a couple of weeks ago, though hadn't built it, and just issued "git pull" to update) on my up-to-date Snow Leopard system, ./configure is failing with: -- Performing Test openssl_greater_than_0_9_7 -- Performing Test openssl_greater_than_0_9_7 - Failed CMake Error at cmake/OpenSSLTests.cmake:41 (message): OpenSSL >= v0.9.7 required Call Stack (most recent call first): CMakeLists.txt:165 (include) However, "openssl version" reports: OpenSSL 1.0.0d 8 Feb 2011 Perhaps there's some sort of search path confusion going on? In general, I have now idea how to fix this - please advise. Vern From jsiwek at ncsa.illinois.edu Tue Jul 19 10:01:16 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Tue, 19 Jul 2011 12:01:16 -0500 (CDT) Subject: [Bro-Dev] build problem on Snow Leopard In-Reply-To: <20110719162826.1364036A37D@taffy.ICSI.Berkeley.EDU> Message-ID: > -- Performing Test openssl_greater_than_0_9_7 > -- Performing Test openssl_greater_than_0_9_7 - Failed > CMake Error at cmake/OpenSSLTests.cmake:41 (message): > OpenSSL >= v0.9.7 required > Call Stack (most recent call first): > CMakeLists.txt:165 (include) Higher up in the ./configure output it should tell which libs it found, e.g: -- Found OpenSSL: /usr/lib/libssl.dylib;/usr/lib/libcrypto.dylib Can you show me what that says for you and also send the contents of build/CMakeFiles/CMakeError.log ? > However, "openssl version" reports: > > OpenSSL 1.0.0d 8 Feb 2011 > > Perhaps there's some sort of search path confusion going on? Yeah, I don't think that's the openssl that comes with snow leopard, so there's probably a mismatch going on somewhere. > In general, I have now idea how to fix this - please advise. There's a --with-openssl option you can give to configure to try and force which installation of openssl you want. But generally this should work automatically on Mac for common package managers (homebrew, macports, and fink), so there's probably something in the CMake scripts I need to fix again. - Jon From vern at icir.org Tue Jul 19 10:51:48 2011 From: vern at icir.org (Vern Paxson) Date: Tue, 19 Jul 2011 10:51:48 -0700 Subject: [Bro-Dev] controlling what's loaded Message-ID: <20110719175148.9059B36A382@taffy.ICSI.Berkeley.EDU> When I run Bro 2.0 with just the script "protocols/dns", it pulls in (among other things) frameworks/packet-filter/netstats.bro , which strikes me as definitely something optional rather than necessary. If I wanted to suppress that behavior, how would I do it? Vern From vern at icir.org Tue Jul 19 10:55:06 2011 From: vern at icir.org (Vern Paxson) Date: Tue, 19 Jul 2011 10:55:06 -0700 Subject: [Bro-Dev] question re policy.old/ Message-ID: <20110719175506.7FA2E36A37D@taffy.ICSI.Berkeley.EDU> Is the point of this directory to just preserve the old policy scripts for purposes of documentation, or is it to provide users with something that works in the former style? It's missing the old bro.init, BTW. Vern From bro at tracker.bro-ids.org Tue Jul 19 11:17:50 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 19 Jul 2011 18:17:50 -0000 Subject: [Bro-Dev] #495: outdated comment in PktSrc.cc Message-ID: <046.bb6220ae577a0cfb473f2f7f88c8a87f@tracker.bro-ids.org> #495: outdated comment in PktSrc.cc ---------------------+----------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: Low | Milestone: Component: Bro | Version: Keywords: | ---------------------+----------------- The following comment: {{{ // We don't put file sources into non-blocking mode as // otherwise we would not be able to identify the EOF // via next_packet(). }}} refers to the function `next_packet()`, which no longer exists. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Jul 19 11:43:08 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 19 Jul 2011 18:43:08 -0000 Subject: [Bro-Dev] #496: bro -H undocumented Message-ID: <046.b7a7f615494ff19ec5f13223043fefcf@tracker.bro-ids.org> #496: bro -H undocumented ---------------------+----------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: Low | Milestone: Component: Bro | Version: Keywords: | ---------------------+----------------- I wanted to get help output so I ran `bro -H` (misremembering that it's instead `-h`). Bro just sat there waiting for input from stdin, even though `bro -h` indicates there's no such flag as `-H`. Turns out it's an alias for `--save-seeds`. Seems we should either remove the alias (my vote), or make sure it appears in the `-h` output. -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Tue Jul 19 12:24:43 2011 From: seth at icir.org (Seth Hall) Date: Tue, 19 Jul 2011 15:24:43 -0400 Subject: [Bro-Dev] question re policy.old/ In-Reply-To: <20110719175506.7FA2E36A37D@taffy.ICSI.Berkeley.EDU> References: <20110719175506.7FA2E36A37D@taffy.ICSI.Berkeley.EDU> Message-ID: <10148494-BDB5-4D94-B108-A0997FD2E4A9@icir.org> On Jul 19, 2011, at 1:55 PM, Vern Paxson wrote: > Is the point of this directory to just preserve the old policy scripts > for purposes of documentation, or is it to provide users with something > that works in the former style? I've been using it as an easy way to pull code from the old scripts. My plan was to get rid of it before the release. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Tue Jul 19 12:28:33 2011 From: seth at icir.org (Seth Hall) Date: Tue, 19 Jul 2011 15:28:33 -0400 Subject: [Bro-Dev] controlling what's loaded In-Reply-To: <20110719175148.9059B36A382@taffy.ICSI.Berkeley.EDU> References: <20110719175148.9059B36A382@taffy.ICSI.Berkeley.EDU> Message-ID: <8D557DA7-F8C9-4688-865D-AA72C94671E9@icir.org> On Jul 19, 2011, at 1:51 PM, Vern Paxson wrote: > When I run Bro 2.0 with just the script "protocols/dns", it pulls in (among > other things) frameworks/packet-filter/netstats.bro , which strikes me as > definitely something optional rather than necessary. If I wanted to > suppress that behavior, how would I do it? That's a good point, I think you've identified a place where we don't want a script to be loaded by default. We're including a local.bro script with Bro now too that's intended to be edited locally after installation as a "recommended" load set so we can do recommended loads there and not make them load by default anymore. I'll change that now and keep the list of odd-to-load-by-default scripts coming. :) My comment to Robin yesterday was that you can think of the new site/local.bro script like the old brolite script except that it's meant to be edited now. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Tue Jul 19 13:35:20 2011 From: seth at icir.org (Seth Hall) Date: Tue, 19 Jul 2011 16:35:20 -0400 Subject: [Bro-Dev] notice actions Message-ID: I'm trying to come up with a way to add the code to make it possible to determine which notice policy item added an action to a notice and I'm running into some trouble. My initial thought was to make the set of notices actions turn into a table yielding a set of counts. Like this... ## The actions that are to be applied to this notice. The set[count] ## is to indicate which :bro:id:`Notice::policy` items ## triggered the action being added to the notice. actions: table[Notice::Action] of set[count] &log &optional; The problem with this is that tables aren't loggable. I also though about using a multiply keyed index for the set, but that doesn't work either because I need to be able to check for action membership in each of the action plugins (that actually does the action). I'm just not coming up with any good solutions unfortunately and I'm getting the feeling we're going to have to hack this in. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From vern at icir.org Tue Jul 19 14:51:50 2011 From: vern at icir.org (Vern Paxson) Date: Tue, 19 Jul 2011 14:51:50 -0700 Subject: [Bro-Dev] what are .rst files? Message-ID: <20110719215150.C32E636A38C@taffy.ICSI.Berkeley.EDU> While debugging Bro, during one of the runs it created: -rw-r--r-- 1 vern admin 0 Jul 19 11:39 ^.rst -rw-r--r-- 1 vern admin 0 Jul 19 11:39 bro.bif.rst -rw-r--r-- 1 vern admin 0 Jul 19 11:39 bro.init.rst -rw-r--r-- 1 vern admin 0 Jul 19 11:39 const.bif.rst -rw-r--r-- 1 vern admin 0 Jul 19 11:39 event.bif.rst -rw-r--r-- 1 vern admin 0 Jul 19 11:39 frameworks^logging^__load__.rst -rw-r--r-- 1 vern admin 0 Jul 19 11:39 frameworks^logging^base.rst -rw-r--r-- 1 vern admin 0 Jul 19 11:39 frameworks^logging^plugins^ascii.rst -rw-r--r-- 1 vern admin 0 Jul 19 11:39 frameworks^notice^__load__.rst -rw-r--r-- 1 vern admin 0 Jul 19 11:39 frameworks^notice^base.rst -rw-r--r-- 1 vern admin 0 Jul 19 11:39 frameworks^notice^weird.rst -rw-r--r-- 1 vern admin 0 Jul 19 11:39 frameworks^packet-filter^__load__.rst -rw-r--r-- 1 vern admin 0 Jul 19 11:39 frameworks^packet-filter^base.rst -rw-r--r-- 1 vern admin 0 Jul 19 11:39 frameworks^packet-filter^netstats.rst -rw-r--r-- 1 vern admin 0 Jul 19 11:39 logging.bif.rst -rw-r--r-- 1 vern admin 0 Jul 19 11:39 reporter.bif.rst -rw-r--r-- 1 vern admin 0 Jul 19 11:39 site.rst -rw-r--r-- 1 vern admin 0 Jul 19 11:39 strings.bif.rst -rw-r--r-- 1 vern admin 0 Jul 19 11:39 types.bif.rst -rw-r--r-- 1 vern admin 0 Jul 19 11:39 utils^conn_ids.rst -rw-r--r-- 1 vern admin 0 Jul 19 11:39 utils^pattern.rst (Note the charmingly named "^.rst"!) What's up with this? Vern From slagell at ncsa.illinois.edu Tue Jul 19 14:53:46 2011 From: slagell at ncsa.illinois.edu (Adam J. Slagell) Date: Tue, 19 Jul 2011 16:53:46 -0500 Subject: [Bro-Dev] what are .rst files? In-Reply-To: <20110719215150.C32E636A38C@taffy.ICSI.Berkeley.EDU> References: <20110719215150.C32E636A38C@taffy.ICSI.Berkeley.EDU> Message-ID: Output from the Documentation framework? On Jul 19, 2011, at 4:51 PM, Vern Paxson wrote: > While debugging Bro, during one of the runs it created: > > -rw-r--r-- 1 vern admin 0 Jul 19 11:39 ^.rst > -rw-r--r-- 1 vern admin 0 Jul 19 11:39 bro.bif.rst > -rw-r--r-- 1 vern admin 0 Jul 19 11:39 bro.init.rst > -rw-r--r-- 1 vern admin 0 Jul 19 11:39 const.bif.rst > -rw-r--r-- 1 vern admin 0 Jul 19 11:39 event.bif.rst > -rw-r--r-- 1 vern admin 0 Jul 19 11:39 frameworks^logging^__load__.rst > -rw-r--r-- 1 vern admin 0 Jul 19 11:39 frameworks^logging^base.rst > -rw-r--r-- 1 vern admin 0 Jul 19 11:39 frameworks^logging^plugins^ascii.rst > -rw-r--r-- 1 vern admin 0 Jul 19 11:39 frameworks^notice^__load__.rst > -rw-r--r-- 1 vern admin 0 Jul 19 11:39 frameworks^notice^base.rst > -rw-r--r-- 1 vern admin 0 Jul 19 11:39 frameworks^notice^weird.rst > -rw-r--r-- 1 vern admin 0 Jul 19 11:39 frameworks^packet-filter^__load__.rst > -rw-r--r-- 1 vern admin 0 Jul 19 11:39 frameworks^packet-filter^base.rst > -rw-r--r-- 1 vern admin 0 Jul 19 11:39 frameworks^packet-filter^netstats.rst > -rw-r--r-- 1 vern admin 0 Jul 19 11:39 logging.bif.rst > -rw-r--r-- 1 vern admin 0 Jul 19 11:39 reporter.bif.rst > -rw-r--r-- 1 vern admin 0 Jul 19 11:39 site.rst > -rw-r--r-- 1 vern admin 0 Jul 19 11:39 strings.bif.rst > -rw-r--r-- 1 vern admin 0 Jul 19 11:39 types.bif.rst > -rw-r--r-- 1 vern admin 0 Jul 19 11:39 utils^conn_ids.rst > -rw-r--r-- 1 vern admin 0 Jul 19 11:39 utils^pattern.rst > > (Note the charmingly named "^.rst"!) > > What's up with this? > > Vern > _______________________________________________ > bro-dev mailing list > bro-dev at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > ------ Adam J. Slagell, CISSP Sr. Security Engineer National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info 217.244.8965 From vern at icir.org Tue Jul 19 14:55:48 2011 From: vern at icir.org (Vern Paxson) Date: Tue, 19 Jul 2011 14:55:48 -0700 Subject: [Bro-Dev] what are .rst files? In-Reply-To: (Tue, 19 Jul 2011 16:53:46 CDT). Message-ID: <20110719215548.9194B36A38B@taffy.ICSI.Berkeley.EDU> > Output from the Documentation framework? Well, if you want to call zero-length files "documentation" :-). Yeah, I must have tripped across one of Bro's flags to do that. That said, (1) why are these files empty, and (2) why does one of them have the name ^.rst ? Vern From robin at icir.org Tue Jul 19 14:58:19 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 19 Jul 2011 14:58:19 -0700 Subject: [Bro-Dev] notice actions In-Reply-To: References: Message-ID: <20110719215819.GS63143@icir.org> On Tue, Jul 19, 2011 at 16:35 -0400, you wrote: > I'm trying to come up with a way to add the code to make it possible > to determine which notice policy item added an action to a notice and > I'm running into some trouble. Is it sufficient to record just the notice item but not the action type? Then you could do just set[count]. Or you do the table but manually turn it into a string for logging. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From jsiwek at ncsa.illinois.edu Tue Jul 19 15:03:37 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Tue, 19 Jul 2011 17:03:37 -0500 (CDT) Subject: [Bro-Dev] what are .rst files? In-Reply-To: <20110719215548.9194B36A38B@taffy.ICSI.Berkeley.EDU> Message-ID: <80648120-caf9-48a5-92ae-5466d6495b2b@tangent.ncsa.illinois.edu> > > Output from the Documentation framework? Yes. (the command line args to toggle it are -Z and --doc-scripts) > (1) why are these files empty, because bro crashed at a point point during parsing, but before writing them? > and (2) why does one of them have the name ^.rst The part is a bug, it should be called just .rst (meaning generated documentation for scripting given to bro on standard input) - Jon From bro at tracker.bro-ids.org Tue Jul 19 15:07:49 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 19 Jul 2011 22:07:49 -0000 Subject: [Bro-Dev] #497: bro -Z generates a file with "internal error" in its name Message-ID: <046.949382cf5955c7a7219675b37f0cad41@tracker.bro-ids.org> #497: bro -Z generates a file with "internal error" in its name ---------------------+----------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: Low | Milestone: Component: Bro | Version: Keywords: | ---------------------+----------------- If you issue `echo | bro -Z` then one of the generated files is `^.rst` . -- Ticket URL: Bro Tracker Bro Issue Tracker From vern at icir.org Tue Jul 19 15:08:07 2011 From: vern at icir.org (Vern Paxson) Date: Tue, 19 Jul 2011 15:08:07 -0700 Subject: [Bro-Dev] what are .rst files? In-Reply-To: <80648120-caf9-48a5-92ae-5466d6495b2b@tangent.ncsa.illinois.edu> (Tue, 19 Jul 2011 17:03:37 CDT). Message-ID: <20110719220808.0863F36A38C@taffy.ICSI.Berkeley.EDU> > because bro crashed at a point point during parsing, but before writing them? Ah - yeah, pretty much. I reproduced the behavior - running bro -Z and then issuing ^C to terminate it results in zero-length files. If instead I provide it with any valid input, then I get the generated files. > The part is a bug, it should be called just .rst > (meaning generated documentation for scripting given to bro on standard input) Okay, I've filed a ticket on it. Vern From bro at tracker.bro-ids.org Tue Jul 19 15:10:50 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 19 Jul 2011 22:10:50 -0000 Subject: [Bro-Dev] #497: bro -Z generates a file with "internal error" in its name In-Reply-To: <046.949382cf5955c7a7219675b37f0cad41@tracker.bro-ids.org> References: <046.949382cf5955c7a7219675b37f0cad41@tracker.bro-ids.org> Message-ID: <061.150db5f00b37361fea3eeabb476c578e@tracker.bro-ids.org> #497: bro -Z generates a file with "internal error" in its name ----------------------+---------------------- Reporter: vern | Owner: jsiwek Type: Problem | Status: assigned Priority: Low | Milestone: Component: Bro | Version: Resolution: | Keywords: ----------------------+---------------------- Changes (by jsiwek): * owner: => jsiwek * status: new => assigned -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Jul 19 17:26:44 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 20 Jul 2011 00:26:44 -0000 Subject: [Bro-Dev] #498: Efficiency problem with remote log flushing Message-ID: <047.837129432a53b3b6b387597dd321a7c9@tracker.bro-ids.org> #498: Efficiency problem with remote log flushing ------------------------+--------------------- Reporter: robin | Type: Problem Status: new | Priority: Normal Milestone: Bro1.6 | Component: Bro Version: git/master | Keywords: preview ------------------------+--------------------- There's FlushLogBuffer() call at the end of RemoteSerializer::SendLogWrite() that's called for every log write, which is inefficient and shouldn't be necessary. However, if I remove it, the remote-logging unit test fails, presumably because now the final flush message that happens just before the connection is torn down doesn't get through anymore. Need to fix that. -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Tue Jul 19 17:31:28 2011 From: seth at icir.org (Seth Hall) Date: Tue, 19 Jul 2011 20:31:28 -0400 Subject: [Bro-Dev] notice actions In-Reply-To: <20110719215819.GS63143@icir.org> References: <20110719215819.GS63143@icir.org> Message-ID: On Jul 19, 2011, at 5:58 PM, Robin Sommer wrote: > Is it sufficient to record just the notice item but not the action > type? Then you could do just set[count]. You wouldn't be able to do the membership checks anymore. They're done like this now... if ( ACTION_EMAIL in n$actions ) email_notice_to(n, mail_dest, T); > Or you do the table but manually turn it into a string for logging. Yeah, I thought of that. It just feels really hacky and I've trying to avoid things that feel hacky. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From robin at icir.org Tue Jul 19 17:44:21 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 19 Jul 2011 17:44:21 -0700 Subject: [Bro-Dev] Failing tests Message-ID: <20110720004421.GZ63143@icir.org> These fail for me currently and I've lost track of whether that's known/expected? doc.autogen-reST-all ... failed core.load-pkg ... failed core.load-unload ... failed policy.misc.check-test-all ... failed policy.misc.testing-coverage ... failed policy.misc.loaded-scripts ... failed policy.protocols.http.http-header-crlf ... failed policy.protocols.http.http-pipelining-and-md5 ... failed 8 tests failed, 2 skipped This is with all pending stuff merged, push will come in a bit. All the external/* tests still fail as well. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Tue Jul 19 17:45:15 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 20 Jul 2011 00:45:15 -0000 Subject: [Bro-Dev] #475: Update Broccoli Test Scripts In-Reply-To: <048.a284abbfb80e3c325b0a34424a7f512f@tracker.bro-ids.org> References: <048.a284abbfb80e3c325b0a34424a7f512f@tracker.bro-ids.org> Message-ID: <063.89ea856837e8a0b0db66d6f0a0a944bd@tracker.bro-ids.org> #475: Update Broccoli Test Scripts -----------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Merge Request | Status: closed Priority: Low | Milestone: Bro1.6 Component: Broccoli | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by robin): * status: assigned => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Jul 19 17:46:54 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 20 Jul 2011 00:46:54 -0000 Subject: [Bro-Dev] #495: outdated comment in PktSrc.cc In-Reply-To: <046.bb6220ae577a0cfb473f2f7f88c8a87f@tracker.bro-ids.org> References: <046.bb6220ae577a0cfb473f2f7f88c8a87f@tracker.bro-ids.org> Message-ID: <061.c86dae1fe4537ad9dee663ea155242c0@tracker.bro-ids.org> #495: outdated comment in PktSrc.cc -----------------------------+-------------------- Reporter: vern | Owner: Type: Problem | Status: closed Priority: Low | Milestone: Component: Bro | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Jul 19 17:58:51 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 20 Jul 2011 00:58:51 -0000 Subject: [Bro-Dev] #494: topic/remove-tabs - Tabs in broccoli-python may have caused trouble. In-Reply-To: <046.8ac9b3e0460e4ca13a0af895fb855b81@tracker.bro-ids.org> References: <046.8ac9b3e0460e4ca13a0af895fb855b81@tracker.bro-ids.org> Message-ID: <061.c06f106fb945eb9fc81700589749c74a@tracker.bro-ids.org> #494: topic/remove-tabs - Tabs in broccoli-python may have caused trouble. ------------------------------+------------------------ Reporter: seth | Owner: robin Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: broccoli-python | Version: git/master Resolution: Solved/Applied | Keywords: ------------------------------+------------------------ Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Tue Jul 19 17:59:10 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 19 Jul 2011 17:59:10 -0700 Subject: [Bro-Dev] Trac commands via git (Re: [Bro-Commits] [git/bro] master diff: Removing undocumented -H) command line flag. (6253c7c) In-Reply-To: <201107200055.p6K0t6fN006586@bro-ids.icir.org> References: <201107200055.p6K0t6fN006586@bro-ids.icir.org> Message-ID: <20110720005910.GA13950@icir.org> On Tue, Jul 19, 2011 at 17:55 -0700, I wrote: > > Closes #496. > Giving Trac commands in git log messages doesn't seem to work? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Tue Jul 19 18:43:24 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 20 Jul 2011 01:43:24 -0000 Subject: [Bro-Dev] #499: test ticket Message-ID: <046.4cc7dd275323d04d2b9f13fe04edb744@tracker.bro-ids.org> #499: test ticket ---------------------+----------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: Keywords: | ---------------------+----------------- testing... -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Jul 19 18:54:28 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 20 Jul 2011 01:54:28 -0000 Subject: [Bro-Dev] #499: test ticket In-Reply-To: <046.4cc7dd275323d04d2b9f13fe04edb744@tracker.bro-ids.org> References: <046.4cc7dd275323d04d2b9f13fe04edb744@tracker.bro-ids.org> Message-ID: <061.6b81852b44a2667752931e3e68847612@tracker.bro-ids.org> #499: test ticket ----------------------+----------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: Resolution: | Keywords: ----------------------+----------------- Comment (by Seth Hall ): In [cedbd7db3548b5b89510868ddab825f396e8fc79/bro]: {{{ #!CommitTicketReference repository="bro" revision="cedbd7db3548b5b89510868ddab825f396e8fc79" Test. Closes #499 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Tue Jul 19 19:00:27 2011 From: seth at icir.org (Seth Hall) Date: Tue, 19 Jul 2011 22:00:27 -0400 Subject: [Bro-Dev] Trac commands via git (Re: [Bro-Commits] [git/bro] master diff: Removing undocumented -H) command line flag. (6253c7c) In-Reply-To: <20110720005910.GA13950@icir.org> References: <201107200055.p6K0t6fN006586@bro-ids.icir.org> <20110720005910.GA13950@icir.org> Message-ID: On Jul 19, 2011, at 8:59 PM, Robin Sommer wrote: > Giving Trac commands in git log messages doesn't seem to work? Fixed. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From jsiwek at ncsa.illinois.edu Tue Jul 19 19:14:26 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Tue, 19 Jul 2011 21:14:26 -0500 (CDT) Subject: [Bro-Dev] Failing tests In-Reply-To: <20110720004421.GZ63143@icir.org> Message-ID: <717307275.4570.1311128066992.JavaMail.root@zimbra-1.ncsa.uiuc.edu> > doc.autogen-reST-all ... failed Expected, I'm kind of waiting until the policy script re-organization settles down to do something with it. > core.load-pkg ... failed > core.load-unload ... failed > policy.misc.check-test-all ... failed > policy.misc.testing-coverage ... failed > policy.misc.loaded-scripts ... failed I don't think there's any reason for these to fail except that baselines haven't been updated or some @load in the test needs to be renamed because it got moved around. > policy.protocols.http.http-header-crlf ... failed > policy.protocols.http.http-pipelining-and-md5 ... failed Not sure about these, haven't looked at them before. > All the external/* tests still fail as well. I don't think those were being checked/updated in sync with script updates. - Jon From seth at icir.org Tue Jul 19 19:36:50 2011 From: seth at icir.org (Seth Hall) Date: Tue, 19 Jul 2011 22:36:50 -0400 Subject: [Bro-Dev] Failing tests In-Reply-To: <717307275.4570.1311128066992.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <717307275.4570.1311128066992.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <283E0212-44CF-4E9B-B3A5-85E2EBB3EF81@icir.org> On Jul 19, 2011, at 10:14 PM, Jonathan Siwek wrote: >> core.load-pkg ... failed >> core.load-unload ... failed >> policy.misc.check-test-all ... failed >> policy.misc.testing-coverage ... failed >> policy.misc.loaded-scripts ... failed Hm, I thought I fixed most of those earlier. I didn't do the check-test-all one though because that one seems a bit more involved. >> policy.protocols.http.http-header-crlf ... failed >> policy.protocols.http.http-pipelining-and-md5 ... failed > > Not sure about these, haven't looked at them before. Hm, I thought I fixed policy.protocols.http.http-pipelining-and-md5 earlier today. I'm not sure what's wrong with the other one though .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Tue Jul 19 22:23:19 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 20 Jul 2011 05:23:19 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: carpinteyrotlp Message-ID: <20110720052317.5AEB7369FC9@taffy.ICSI.Berkeley.EDU> New user registration for user carpinteyrotlp -- Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jul 20 08:17:58 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 20 Jul 2011 15:17:58 -0000 Subject: [Bro-Dev] #499: test ticket In-Reply-To: <046.4cc7dd275323d04d2b9f13fe04edb744@tracker.bro-ids.org> References: <046.4cc7dd275323d04d2b9f13fe04edb744@tracker.bro-ids.org> Message-ID: <061.d6205690e315c4728001716ac95bdde3@tracker.bro-ids.org> #499: test ticket -----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Component: Bro | Version: Resolution: Rejected | Keywords: -----------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Rejected Comment: Hm, the comment made it here but it didn't close. It's possibly due to the fact that apparently my SSH config was wrong (since fixed) and had the wrong email address. I'm closing the ticket manually now. -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Wed Jul 20 10:29:57 2011 From: gregor at icir.org (Gregor Maier) Date: Wed, 20 Jul 2011 10:29:57 -0700 Subject: [Bro-Dev] build problem on Snow Leopard In-Reply-To: References: Message-ID: <4E271095.6030004@icir.org> >> Perhaps there's some sort of search path confusion going on? > > Yeah, I don't think that's the openssl that comes with snow leopard, > so there's probably a mismatch going on somewhere. Yep. Snow Leopard ships with an openssl < 1.0 but macports has one > 1.0. > But generally this should work automatically on Mac for common package > managers (homebrew, macports, and fink), so there's probably something > in the CMake scripts I need to fix again. FWIW, I tried it on my Snow Leopard with macports and it worked for me: -- Found OpenSSL: /opt/local/lib/libssl.dylib;/opt/local/lib/libcrypto.dylib $ openssl version OpenSSL 1.0.0d 8 Feb 2011 -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Wed Jul 20 13:30:56 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 20 Jul 2011 20:30:56 -0000 Subject: [Bro-Dev] #469: Core IRC analyzer needs to provide is_orig field. In-Reply-To: <046.ead7a0f871260990c028a605e157ef4b@tracker.bro-ids.org> References: <046.ead7a0f871260990c028a605e157ef4b@tracker.bro-ids.org> Message-ID: <061.a1e46915e242dc10fbe31c8bfe5860a6@tracker.bro-ids.org> #469: Core IRC analyzer needs to provide is_orig field. ----------------------------+------------------------ Reporter: seth | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: preview ----------------------------+------------------------ Changes (by jsiwek): * type: Problem => Merge Request Comment: This is implemented in `topic/jsiwek/irc-orig`, w/ some other fixes related to the DCC events. I just added the `is_orig` argument to all irc core-generated events for now instead of re-implementing the events in bro scripting. Doing that at a later time should be possible in a way that won't break user's code that rely on the events. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jul 20 14:16:16 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 20 Jul 2011 21:16:16 -0000 Subject: [Bro-Dev] #497: bro -Z generates a file with "internal error" in its name In-Reply-To: <046.949382cf5955c7a7219675b37f0cad41@tracker.bro-ids.org> References: <046.949382cf5955c7a7219675b37f0cad41@tracker.bro-ids.org> Message-ID: <061.05e0cffd63855d148cd9ba0d86939fc5@tracker.bro-ids.org> #497: bro -Z generates a file with "internal error" in its name ----------------------+-------------------- Reporter: vern | Owner: jsiwek Type: Problem | Status: closed Priority: Low | Milestone: Component: Bro | Version: Resolution: fixed | Keywords: ----------------------+-------------------- Changes (by jsiwek): * status: assigned => closed * resolution: => fixed Comment: In [54c14025d9fe3cb92021439c6f0f3cff24dc58b0/bro]: {{{ #!CommitTicketReference repository="bro" revision="54c14025d9fe3cb92021439c6f0f3cff24dc58b0" Fix reST file name associated w/ stdin when in doc mode (closes #497) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jul 20 15:26:54 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 20 Jul 2011 22:26:54 -0000 Subject: [Bro-Dev] #500: PktSrc::Statistics generates bogus stats when reading a trace Message-ID: <046.d0332ebd9070627807d30aec450eea2e@tracker.bro-ids.org> #500: PktSrc::Statistics generates bogus stats when reading a trace ---------------------+-------------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: Preview | ---------------------+-------------------- `PktSrc::Statistics` winds up returning values from uninitialized memory when `reading_traces` is true. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jul 20 15:30:51 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 20 Jul 2011 22:30:51 -0000 Subject: [Bro-Dev] #501: netstats.bro reports drop stats when reading traces Message-ID: <046.d7da2e6c1f8c1eb15a4e7918eceb2bb8@tracker.bro-ids.org> #501: netstats.bro reports drop stats when reading traces ---------------------+-------------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: | ---------------------+-------------------- `netstats.bro` winds up reporting drop statistics even when reading trace files. Bro 1.5 had logic in it to avoid this. -- Ticket URL: Bro Tracker Bro Issue Tracker From vern at icir.org Wed Jul 20 15:43:05 2011 From: vern at icir.org (Vern Paxson) Date: Wed, 20 Jul 2011 15:43:05 -0700 Subject: [Bro-Dev] "git diff" questions Message-ID: <20110720224305.E40BA36A38F@taffy.ICSI.Berkeley.EDU> On a checkout for which I haven't touched aux/broccoli, if I issue "git diff" it tells me: diff --git a/aux/broccoli b/aux/broccoli index 9866a00..8843da5 160000 --- a/aux/broccoli +++ b/aux/broccoli @@ -1 +1 @@ -Subproject commit 9866a00e78b088315e632960aaa230635bca326a +Subproject commit 8843da57dc8aee433550727dcbd1199824ca9da4 (1) Why is it finding a difference? (2) How do I get it to show me what the difference actually is? Vern From vern at icir.org Wed Jul 20 15:54:45 2011 From: vern at icir.org (Vern Paxson) Date: Wed, 20 Jul 2011 15:54:45 -0700 Subject: [Bro-Dev] build problem on Snow Leopard In-Reply-To: (Tue, 19 Jul 2011 12:01:16 CDT). Message-ID: <20110720225445.BCCE236A38F@taffy.ICSI.Berkeley.EDU> > Can you show me what that says for you and also send the contents of > build/CMakeFiles/CMakeError.log ? Ah, now that I know to look there, I see the problem. It looks like the OpenSSL setup I cloned a few days ago from my previous laptop has an architecture mismatch for my new laptop. Time to rebuild. If you don't hear from me (other than howls of dismay over MacPorts clunkiness ;-), then that was indeed the problem. Vern From bro at tracker.bro-ids.org Wed Jul 20 17:09:14 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 21 Jul 2011 00:09:14 -0000 Subject: [Bro-Dev] #469: Core IRC analyzer needs to provide is_orig field. In-Reply-To: <046.ead7a0f871260990c028a605e157ef4b@tracker.bro-ids.org> References: <046.ead7a0f871260990c028a605e157ef4b@tracker.bro-ids.org> Message-ID: <061.79065918fc5aef0aedff8653e63704d1@tracker.bro-ids.org> #469: Core IRC analyzer needs to provide is_orig field. ----------------------------+------------------------ Reporter: seth | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: preview ----------------------------+------------------------ Comment (by robin): Merged, but I noticed that the irc_enter event also disappeared. Is that intentional? If yes, just close this ticket. Also, the IRC unit tests were failing. The output I got looked "more correct" so I've updated the baseline, please check the diff. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jul 20 17:13:41 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 21 Jul 2011 00:13:41 -0000 Subject: [Bro-Dev] #469: Core IRC analyzer needs to provide is_orig field. In-Reply-To: <046.ead7a0f871260990c028a605e157ef4b@tracker.bro-ids.org> References: <046.ead7a0f871260990c028a605e157ef4b@tracker.bro-ids.org> Message-ID: <061.0c80b6595d250464bb6b90619bdde758@tracker.bro-ids.org> #469: Core IRC analyzer needs to provide is_orig field. ----------------------------+------------------------ Reporter: seth | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: preview ----------------------------+------------------------ Comment (by robin): In [2892026201252d25e6770eb5b0e0dce481f80984/bro]: {{{ #!CommitTicketReference repository="bro" revision="2892026201252d25e6770eb5b0e0dce481f80984" Merge remote-tracking branch 'origin/topic/jsiwek/irc-orig' * origin/topic/jsiwek/irc-orig: Shorten what's displayed in the IRC's log mime_type column for DCC transfers Add IRC unit tests. Small tweak to IRC event handlder priorities Fix IRC analyzer supplying wrong type to irc_dcc_message event. Changes to IRC analyzer and events (addresses #469). }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Wed Jul 20 17:33:35 2011 From: gregor at icir.org (Gregor Maier) Date: Wed, 20 Jul 2011 17:33:35 -0700 Subject: [Bro-Dev] "git diff" questions In-Reply-To: <20110720224305.E40BA36A38F@taffy.ICSI.Berkeley.EDU> References: <20110720224305.E40BA36A38F@taffy.ICSI.Berkeley.EDU> Message-ID: <4E2773DF.8010206@icir.org> This is a, rather annoying, effect of using git submodules. aux/broccoli just points to a particular commit in the broccoli repository. Different topic branches in the main bro repository can point to different commits of submodules. This mostly happens if you are working / looking at a branch that was branched off master a while ago and the submodule pointer in master has then been updated. > (1) Why is it finding a difference? For some reason that I don't understand, a git checkout does not check out the appropriate commit of the submodules. It rather leaves the submodules untouched. In order to also do a checkout of the submodules you need to do a git submodule update --recursive On 7/20/11 15:43 , Vern Paxson wrote: > On a checkout for which I haven't touched aux/broccoli, if I issue "git diff" > it tells me: > > diff --git a/aux/broccoli b/aux/broccoli > index 9866a00..8843da5 160000 > --- a/aux/broccoli > +++ b/aux/broccoli > @@ -1 +1 @@ > -Subproject commit 9866a00e78b088315e632960aaa230635bca326a > +Subproject commit 8843da57dc8aee433550727dcbd1199824ca9da4 Assuming that you were in "master" and did a check-out of "topic/foobar" the above diff indicates that master uses commit 9866a00.... of broccoli but that foobar uses commit 8843da5.... of broccoli hth Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From vern at icir.org Wed Jul 20 17:38:45 2011 From: vern at icir.org (Vern Paxson) Date: Wed, 20 Jul 2011 17:38:45 -0700 Subject: [Bro-Dev] "git diff" questions In-Reply-To: <4E2773DF.8010206@icir.org> (Wed, 20 Jul 2011 17:33:35 PDT). Message-ID: <20110721003845.B17F636A00A@taffy.ICSI.Berkeley.EDU> Thanks for the explanation. Two comments/questions: > git submodule update --recursive This doesn't work: Usage: git submodule [--quiet] [--cached] [add [-b branch] ]|[status|init|update [-i|--init] [-N|--no-fetch]|summary [-n|--summary-limit ] []] [--] [...]|[foreach ]|[sync [--] [...]] (I'm running git 1.6.3.2.) > Assuming that you were in "master" and did a check-out of "topic/foobar" All I did was check-out "master", I'm pretty sure. (Is there a command that will tell me how I wound up with the tree I have?) Vern From bro at tracker.bro-ids.org Wed Jul 20 17:44:12 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 21 Jul 2011 00:44:12 -0000 Subject: [Bro-Dev] #502: scan.l can free a static string Message-ID: <046.5f0edf90368eba5d72d1697dd5206b02@tracker.bro-ids.org> #502: scan.l can free a static string ---------------------+-------------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: | ---------------------+-------------------- If I run bro with no arguments, my Snow Leopard OSX system flags a bad malloc operation at line 630 of `scan.l`: {{{ delete [] bropath_subpath; }}} This occurs because bropath_subpath at that point is still "". -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jul 20 17:46:52 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 21 Jul 2011 00:46:52 -0000 Subject: [Bro-Dev] #503: Update broctl README Message-ID: <047.1a0db4ec338e9735ce78c840cd80bbc3@tracker.bro-ids.org> #503: Update broctl README ------------------------+------------------------ Reporter: robin | Owner: robin Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: git/master Keywords: preview | ------------------------+------------------------ In particular the standalone vs cluster discussion, but generally everything that's not true anymore ... -- Ticket URL: Bro Tracker Bro Issue Tracker From vern at icir.org Wed Jul 20 17:48:53 2011 From: vern at icir.org (Vern Paxson) Date: Wed, 20 Jul 2011 17:48:53 -0700 Subject: [Bro-Dev] "git diff" questions In-Reply-To: <20110721003845.B17F636A00A@taffy.ICSI.Berkeley.EDU> (Wed, 20 Jul 2011 17:38:45 PDT). Message-ID: <20110721004853.80E3B36A00A@taffy.ICSI.Berkeley.EDU> > (I'm running git 1.6.3.2.) Hmmm, so that was old crude due to my laptop upgrade. I'm now installing git 1.7 and will try that. Vern From gregor at icir.org Wed Jul 20 17:49:03 2011 From: gregor at icir.org (Gregor Maier) Date: Wed, 20 Jul 2011 17:49:03 -0700 Subject: [Bro-Dev] "git diff" questions In-Reply-To: <20110721003845.B17F636A00A@taffy.ICSI.Berkeley.EDU> References: <20110721003845.B17F636A00A@taffy.ICSI.Berkeley.EDU> Message-ID: <4E27777F.6070006@icir.org> On 7/20/11 17:38 , Vern Paxson wrote: > Thanks for the explanation. Two comments/questions: > >> git submodule update --recursive > > This doesn't work: > > Usage: git submodule [--quiet] [--cached] [add [-b branch] ]|[status|init|update [-i|--init] [-N|--no-fetch]|summary [-n|--summary-limit] []] [--] [...]|[foreach]|[sync [--] [...]] > > (I'm running git 1.6.3.2.) Hmm. I'm running 1.7.5.4. I guess that added the recursive then.... You can probably use: git submodule update git submodule foreach git submodule update (IIRC the broccoli repository has submodules of its own, so you need to either to it recursively or do it "by hand" using the submodule foreach) >> Assuming that you were in "master" and did a check-out of "topic/foobar" > > All I did was check-out "master", I'm pretty sure. (Is there a command > that will tell me how I wound up with the tree I have?) > Unfortunately not. Since in theory it should not matter how you wound up with the tree. The submodules just bust this nice theory :-( What could also have happend is that you cloned (and checked out) master a while ago and then later updated it with a git pull/fetch. If the submodule pointers in master changed in the meantime you would also get this notice. cu gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Wed Jul 20 17:50:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 21 Jul 2011 00:50:13 -0000 Subject: [Bro-Dev] #502: scan.l can free a static string In-Reply-To: <046.5f0edf90368eba5d72d1697dd5206b02@tracker.bro-ids.org> References: <046.5f0edf90368eba5d72d1697dd5206b02@tracker.bro-ids.org> Message-ID: <061.5c9ad820d62407ee83ac7f1c7186c7aa@tracker.bro-ids.org> #502: scan.l can free a static string ----------------------+-------------------- Reporter: vern | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Invalid | Keywords: ----------------------+-------------------- Changes (by vern): * status: new => closed * resolution: => Invalid Old description: > If I run bro with no arguments, my Snow Leopard OSX system flags a bad > malloc operation at line 630 of `scan.l`: > {{{ > delete [] bropath_subpath; > }}} > This occurs because bropath_subpath at that point is still " error>". New description: If I run bro with no arguments, my Snow Leopard OSX system flags a bad malloc operation at line 630 of `scan.l`: {{{ delete [] bropath_subpath; }}} This occurs because bropath_subpath at that point is still "". Never mind - this was me being in git hell, and having an out-of-date tree even though it didn't appear to be :-(. -- -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Wed Jul 20 17:54:35 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 20 Jul 2011 17:54:35 -0700 Subject: [Bro-Dev] "git diff" questions In-Reply-To: <4E27777F.6070006@icir.org> References: <20110721003845.B17F636A00A@taffy.ICSI.Berkeley.EDU> <4E27777F.6070006@icir.org> Message-ID: <20110721005435.GH62661@icir.org> On Wed, Jul 20, 2011 at 17:49 -0700, you wrote: > What could also have happend is that you cloned (and checked out) master > a while ago and then later updated it with a git pull/fetch. If the > submodule pointers in master changed in the meantime you would also get > this notice. That's my guess actually. I've defined me a macro to get the submodules right, see "git fix-submodules" here: http://www.bro-ids.org/development/process.html#useful-git-macros Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Wed Jul 20 18:23:16 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 21 Jul 2011 01:23:16 -0000 Subject: [Bro-Dev] #469: Core IRC analyzer needs to provide is_orig field. In-Reply-To: <046.ead7a0f871260990c028a605e157ef4b@tracker.bro-ids.org> References: <046.ead7a0f871260990c028a605e157ef4b@tracker.bro-ids.org> Message-ID: <061.40850f3348233e976ef7a6a8cd26d4de@tracker.bro-ids.org> #469: Core IRC analyzer needs to provide is_orig field. -----------------------------+------------------------ Reporter: seth | Owner: jsiwek Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: preview -----------------------------+------------------------ Changes (by jsiwek): * status: assigned => closed * resolution: => Solved/Applied Comment: > Merged, but I noticed that the irc_enter event also disappeared. Is that intentional? If yes, just close this ticket. Sorry, forgot: `irc_enter_message` event wasn't being generated anywhere, so I removed it. > Also, the IRC unit tests were failing. The output I got looked "more correct" so I've updated the baseline, please check the diff. With libmagic 5.04 & 5.07, it's reporting mime type `application/zip`, but you're getting `application/x-zip`. I suppose I need to make the test insensitive to libmagic support entirely since that's an optional dependency. Will do that separately on a fastpath commit later. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jul 20 18:46:27 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 21 Jul 2011 01:46:27 -0000 Subject: [Bro-Dev] #504: topic/script-load-changes Message-ID: <048.569587bed68aec0d9cf35b76f39daad8@tracker.bro-ids.org> #504: topic/script-load-changes ---------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ CHANGES: * Make @load statements recognize relative paths. For example a script can do "@load ./foo" to load a script named foo.bro that lives in the same directory or "@load ../bar" to load a script named bar.bro in the parent directory, even if those directories are not contained in BROPATH. * Reimplementation of the @prefixes statement. (closes #486) Any added prefixes are now used *after* all input files have been parsed to look for a prefixed, flattened version of the input file somewhere in BROPATH and, if found, load it. For example, if "lcl" is in @prefixes, and site.bro is loaded, then a file named "lcl.site.bro" that's in BROPATH would end up being automatically loaded as well. Packages work similarly, e.g. loading "protocols/http" means a file named "lcl.protocols.http.bro" in BROPATH gets loaded automatically. * Fix @unload'd files from generating bro_script_loaded event. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jul 20 18:49:39 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 21 Jul 2011 01:49:39 -0000 Subject: [Bro-Dev] #486: Prefixes don't support directories In-Reply-To: <047.e78f97399b2dd07d57a5512021972454@tracker.bro-ids.org> References: <047.e78f97399b2dd07d57a5512021972454@tracker.bro-ids.org> Message-ID: <062.394a63cbd137951443f0b9bd03e4a4b2@tracker.bro-ids.org> #486: Prefixes don't support directories -----------------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: preview -----------------------------+------------------------ Changes (by jsiwek): * status: new => closed * resolution: => Solved/Applied Comment: Closing this, merge of this change can be tracked in #504. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jul 20 20:15:19 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 21 Jul 2011 03:15:19 -0000 Subject: [Bro-Dev] #469: Core IRC analyzer needs to provide is_orig field. In-Reply-To: <046.ead7a0f871260990c028a605e157ef4b@tracker.bro-ids.org> References: <046.ead7a0f871260990c028a605e157ef4b@tracker.bro-ids.org> Message-ID: <061.fa723a8e14f6fbb69a6dcf4136f754c7@tracker.bro-ids.org> #469: Core IRC analyzer needs to provide is_orig field. -----------------------------+------------------------ Reporter: seth | Owner: jsiwek Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: preview -----------------------------+------------------------ Comment (by seth): > insensitive to libmagic support entirely since that's an optional > dependency. Will do that separately on a fastpath commit later. We actually have the same problem with the http tests too (sometimes it uses libmagic). FTP tests will also be affected by this. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jul 20 20:25:52 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 21 Jul 2011 03:25:52 -0000 Subject: [Bro-Dev] #501: netstats.bro reports drop stats when reading traces In-Reply-To: <046.d7da2e6c1f8c1eb15a4e7918eceb2bb8@tracker.bro-ids.org> References: <046.d7da2e6c1f8c1eb15a4e7918eceb2bb8@tracker.bro-ids.org> Message-ID: <061.dec0a3b96999054e6369e1ac2f56c29c@tracker.bro-ids.org> #501: netstats.bro reports drop stats when reading traces -----------------------------+-------------------- Reporter: vern | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: Fixed with commit 93daf88856c428f4686dc94129975c7c546e59f5. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jul 20 21:20:04 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 21 Jul 2011 04:20:04 -0000 Subject: [Bro-Dev] #505: Invalid Unref crash Message-ID: <048.a3ead8ea8da3e7844a25274cbfce1df6@tracker.bro-ids.org> #505: Invalid Unref crash ------------------------+--------------------- Reporter: gclark | Type: Problem Status: new | Priority: Normal Milestone: | Component: Bro Version: git/master | Keywords: ------------------------+--------------------- Using latest from bro-master: Last few lines of Bro execution trace: {{{ 1311220077.263882 /home/gilbert/Code/bro/policy/frameworks/notice/weird.bro:384 Builtin Function called: network_time() 1311220077.263882 /home/gilbert/Code/bro/policy/frameworks/notice/weird.bro:384 Function return: 1311220077.26388 1311220077.263882 /home/gilbert/Code/bro/policy/utils/conn-ids.bro:23 function called: id_string(id = '[orig_h=204.152.191.37, orig_p=80/tcp, resp_h=212.110.251.3, resp_p=33595/tcp]') 1311220077.263882 /home/gilbert/Code/bro/policy/utils/conn-ids.bro:23 Builtin Function called: fmt(va_args = '%s:%d > %s:%d', vararg0 = '204.152.191.37', vararg1 = '80/tcp', vararg2 = '212.110.251.3', vararg3 = '33595/tcp') 1311220077.263882 /home/gilbert/Code/bro/policy/utils/conn-ids.bro:23 Function return: 204.152.191.37:80 > 212.110.251.3:33595 1311220077.263882 /home/gilbert/Code/bro/policy/utils/conn-ids.bro:23 Function return: 204.152.191.37:80 > 212.110.251.3:33595 1311220077.263882 /home/gilbert/Code/bro/policy/frameworks/notice/weird.bro:359 function called: Weird::report_weird_conn(t = '1311220077.26388', name = 'above_hole_data_without_any_acks', id = '204.152.191.37:80 > 212.110.251.3:33595', addl = '', c = '[id=[orig_h=204.152.191.37, orig_p=80/tcp, resp_h=212.110.251.3, resp_p=33595/tcp], orig=[size=7240, state=3, num_pkts=, num_bytes_ip=], resp=[size=0, state=0, num_pkts=, num_bytes_ip=], start_time=1311220077.2637, duration=0.000185012817382812, service={ }, addl=, hot=0, history=D, uid=vwfIafnipTj, conn=, extract_orig=F, extract_resp=F, dns=, dns_state=, ftp=, http=, http_state=, irc=, mime=, mime_state=, smtp=, smtp_state=, ssh=, ssl=, syslog=]') 1311220077.263882 /home/gilbert/Code/bro/policy/frameworks/notice/weird.bro:310 function called: Weird::report_weird(t = '1311220077.26388', name = 'above_hole_data_without_any_acks', id = '204.152.191.37:80 > 212.110.251.3:33595', have_conn = 'T', addl = '', action = 'WEIRD_FILE', no_log = 'F') 1311220077.263882 /home/gilbert/Code/bro/policy/frameworks/logging/base.bro:188 function called: Log::write(id = 'WEIRD', columns = '[ts=1311220077.26388, uid=vwfIafnipTj, id=[orig_h=204.152.191.37, orig_p=80/tcp, resp_h=212.110.251.3, resp_p=33595/tcp], msg=above_hole_data_without_any_acks, addl=, notice=F]') 1311220077.263882 /home/gilbert/Code/bro/policy/frameworks/logging/base.bro:188 Builtin Function called: Log::__write(id = 'WEIRD', columns = '[ts=1311220077.26388, uid=vwfIafnipTj, id=[orig_h=204.152.191.37, orig_p=80/tcp, resp_h=212.110.251.3, resp_p=33595/tcp], msg=above_hole_data_without_any_acks, addl=, notice=F]') 1311220077.263882 /home/gilbert/Code/bro/policy/frameworks/logging/base.bro:188 Function return: T 1311220077.263882 /home/gilbert/Code/bro/policy/frameworks/logging/base.bro:188 Function return: T 1311220077.461318 /home/gilbert/Code/bro/build/src/event.bif.bro:104 event called: conn_weird(name = 'spontaneous_RST', c = '[id=[orig_h=212.110.251.3, orig_p=113/tcp, resp_h=161.53.178.240, resp_p=50349/tcp], orig=[size=0, state=6, num_pkts=, num_bytes_ip=], resp=[size=0, state=0, num_pkts=, num_bytes_ip=], start_time=1311220077.46132, duration=0.0, service={ }, addl=, hot=0, history=R, uid=0Wn1J3f4jD4, conn=, extract_orig=F, extract_resp=F, dns=, dns_state=, ftp=, http=, http_state=, irc=, mime=, mime_state=, smtp=, smtp_state=, ssh=, ssl=, syslog=]', addl = '') 1311220077.461318 /home/gilbert/Code/bro/policy/frameworks/notice/weird.bro:384 Builtin Function called: network_time() 1311220077.461318 /home/gilbert/Code/bro/policy/frameworks/notice/weird.bro:384 Function return: 1311220077.46132 1311220077.461318 /home/gilbert/Code/bro/policy/utils/conn-ids.bro:23 function called: id_string(id = '[orig_h=212.110.251.3, orig_p=113/tcp, resp_h=161.53.178.240, resp_p=50349/tcp]') 1311220077.461318 /home/gilbert/Code/bro/policy/utils/conn-ids.bro:23 Builtin Function called: fmt(va_args = '%s:%d > %s:%d', vararg0 = '212.110.251.3', vararg1 = '113/tcp', vararg2 = '161.53.178.240', vararg3 = '50349/tcp') 1311220077.461318 /home/gilbert/Code/bro/policy/utils/conn-ids.bro:23 Function return: 212.110.251.3:113 > 161.53.178.240:50349 1311220077.461318 /home/gilbert/Code/bro/policy/utils/conn-ids.bro:23 Function return: 212.110.251.3:113 > 161.53.178.240:50349 1311220077.461318 /home/gilbert/Code/bro/policy/frameworks/notice/weird.bro:359 function called: Weird::report_weird_conn(t = '1311220077.46132', name = 'spontaneous_RST', id = '212.110.251.3:113 > 161.53.178.240:50349', addl = '', c = '[id=[orig_h=212.110.251.3, orig_p=113/tcp, resp_h=161.53.178.240, resp_p=50349/tcp], orig=[size=0, state=6, num_pkts=, num_bytes_ip=], resp=[size=0, state=0, num_pkts=, num_bytes_ip=], start_time=1311220077.46132, duration=0.0, service={ }, addl=, hot=0, history=R, uid=0Wn1J3f4jD4, conn=, extract_orig=F, extract_resp=F, dns=, dns_state=, ftp=, http=, http_state=, irc=, mime=, mime_state=, smtp=, smtp_state=, ssh=, ssl=, syslog=]') 1311220077.461318 /home/gilbert/Code/bro/policy/frameworks/notice/weird.bro:310 function called: Weird::report_weird(t = '1311220077.46132', name = 'spontaneous_RST', id = '212.110.251.3:113 > 161.53.178.240:50349', have_conn = 'T', addl = '', action = 'WEIRD_IGNORE', no_log = 'F') 1311220077.461532 /home/gilbert/Code/bro/build/src/event.bif.bro:104 event called: conn_weird(name = 'connection_originator_SYN_ack', c = '[id=[orig_h=161.53.178.240, orig_p=6667/tcp, resp_h=212.110.251.3, resp_p=59665/tcp], orig=[size=0, state=0, num_pkts=, num_bytes_ip=], resp=[size=0, state=0, num_pkts=, num_bytes_ip=], start_time=1311220077.46094, duration=0.0, service={ }, addl=, hot=0, history=H, uid=hrTdn4R7Yq9, conn=, extract_orig=F, extract_resp=F, dns=, dns_state=, ftp=, http=, http_state=, irc=, mime=, mime_state=, smtp=, smtp_state=, ssh=, ssl=, syslog=]', addl = '') 1311220077.461532 /home/gilbert/Code/bro/policy/frameworks/notice/weird.bro:384 Builtin Function called: network_time() 1311220077.461532 /home/gilbert/Code/bro/policy/frameworks/notice/weird.bro:384 Function return: 1311220077.46153 1311220077.461532 /home/gilbert/Code/bro/policy/utils/conn-ids.bro:23 function called: id_string(id = '[orig_h=161.53.178.240, orig_p=6667/tcp, resp_h=212.110.251.3, resp_p=59665/tcp]') 1311220077.461532 /home/gilbert/Code/bro/policy/utils/conn-ids.bro:23 Builtin Function called: fmt(va_args = '%s:%d > %s:%d', vararg0 = '161.53.178.240', vararg1 = '6667/tcp', vararg2 = '212.110.251.3', vararg3 = '59665/tcp') 1311220077.461532 /home/gilbert/Code/bro/policy/utils/conn-ids.bro:23 Function return: 161.53.178.240:6667 > 212.110.251.3:59665 1311220077.461532 /home/gilbert/Code/bro/policy/utils/conn-ids.bro:23 Function return: 161.53.178.240:6667 > 212.110.251.3:59665 1311220077.461532 /home/gilbert/Code/bro/policy/frameworks/notice/weird.bro:359 function called: Weird::report_weird_conn(t = '1311220077.46153', name = 'connection_originator_SYN_ack', id = '161.53.178.240:6667 > 212.110.251.3:59665', addl = '', c = '[id=[orig_h=161.53.178.240, orig_p=6667/tcp, resp_h=212.110.251.3, resp_p=59665/tcp], orig=[size=0, state=0, num_pkts=, num_bytes_ip=], resp=[size=0, state=0, num_pkts=, num_bytes_ip=], start_time=1311220077.46094, duration=0.0, service={ }, addl=, hot=0, history=H, uid=hrTdn4R7Yq9, conn=, extract_orig=F, extract_resp=F, dns=, dns_state=, ftp=, http=, http_state=, irc=, mime=, mime_state=, smtp=, smtp_state=, ssh=, ssl=, syslog=]') 1311220077.461532 /home/gilbert/Code/bro/policy/frameworks/notice/weird.bro:310 function called: Weird::report_weird(t = '1311220077.46153', name = 'connection_originator_SYN_ack', id = '161.53.178.240:6667 > 212.110.251.3:59665', have_conn = 'T', addl = '', action = 'WEIRD_FILE', no_log = 'F') 1311220077.461532 /home/gilbert/Code/bro/policy/frameworks/logging/base.bro:188 function called: Log::write(id = 'WEIRD', columns = '[ts=1311220077.46153, uid=hrTdn4R7Yq9, id=[orig_h=161.53.178.240, orig_p=6667/tcp, resp_h=212.110.251.3, resp_p=59665/tcp], msg=connection_originator_SYN_ack, addl=, notice=F]') 1311220077.461532 /home/gilbert/Code/bro/policy/frameworks/logging/base.bro:188 Builtin Function called: Log::__write(id = 'WEIRD', columns = '[ts=1311220077.46153, uid=hrTdn4R7Yq9, id=[orig_h=161.53.178.240, orig_p=6667/tcp, resp_h=212.110.251.3, resp_p=59665/tcp], msg=connection_originator_SYN_ack, addl=, notice=F]') 1311220077.461532 /home/gilbert/Code/bro/policy/frameworks/logging/base.bro:188 Function return: T 1311220077.461532 /home/gilbert/Code/bro/policy/frameworks/logging/base.bro:188 Function return: T 1311220077.462050 /home/gilbert/Code/bro/build/src/event.bif.bro:67 event called: protocol_confirmation(c = '[id=[orig_h=212.110.251.3, orig_p=52895/tcp, resp_h=64.18.128.86, resp_p=6667/tcp], orig=[size=38, state=3, num_pkts=, num_bytes_ip=], resp=[size=0, state=0, num_pkts=, num_bytes_ip=], start_time=1311220077.46205, duration=0.0, service={ }, addl=, hot=0, history=D, uid=Y6rpTW6Rgeg, conn=, extract_orig=F, extract_resp=F, dns=, dns_state=, ftp=, http=, http_state=, irc=, mime=, mime_state=, smtp=, smtp_state=, ssh=, ssl=, syslog=]', atype = '19', aid = '8963') 1311220077.462050 /home/gilbert/Code/bro/build/src/event.bif.bro:104 event called: conn_weird(name = 'irc_line_too_short', c = '[id=[orig_h=212.110.251.3, orig_p=52895/tcp, resp_h=64.18.128.86, resp_p=6667/tcp], orig=[size=38, state=3, num_pkts=, num_bytes_ip=], resp=[size=0, state=0, num_pkts=, num_bytes_ip=], start_time=1311220077.46205, duration=0.0, service={ }, addl=, hot=0, history=D, uid=Y6rpTW6Rgeg, conn=, extract_orig=F, extract_resp=F, dns=, dns_state=, ftp=, http=, http_state=, irc=, mime=, mime_state=, smtp=, smtp_state=, ssh=, ssl=, syslog=]', addl = '') 1311220077.462050 /home/gilbert/Code/bro/policy/frameworks/notice/weird.bro:384 Builtin Function called: network_time() 1311220077.462050 /home/gilbert/Code/bro/policy/frameworks/notice/weird.bro:384 Function return: 1311220077.46205 1311220077.462050 /home/gilbert/Code/bro/policy/utils/conn-ids.bro:23 function called: id_string(id = '[orig_h=212.110.251.3, orig_p=52895/tcp, resp_h=64.18.128.86, resp_p=6667/tcp]') 1311220077.462050 /home/gilbert/Code/bro/policy/utils/conn-ids.bro:23 Builtin Function called: fmt(va_args = '%s:%d > %s:%d', vararg0 = '212.110.251.3', vararg1 = '52895/tcp', vararg2 = '64.18.128.86', vararg3 = '6667/tcp') 1311220077.462050 /home/gilbert/Code/bro/policy/utils/conn-ids.bro:23 Function return: 212.110.251.3:52895 > 64.18.128.86:6667 1311220077.462050 /home/gilbert/Code/bro/policy/utils/conn-ids.bro:23 Function return: 212.110.251.3:52895 > 64.18.128.86:6667 1311220077.462050 /home/gilbert/Code/bro/policy/frameworks/notice/weird.bro:359 function called: Weird::report_weird_conn(t = '1311220077.46205', name = 'irc_line_too_short', id = '212.110.251.3:52895 > 64.18.128.86:6667', addl = '', c = '[id=[orig_h=212.110.251.3, orig_p=52895/tcp, resp_h=64.18.128.86, resp_p=6667/tcp], orig=[size=38, state=3, num_pkts=, num_bytes_ip=], resp=[size=0, state=0, num_pkts=, num_bytes_ip=], start_time=1311220077.46205, duration=0.0, service={ }, addl=, hot=0, history=D, uid=Y6rpTW6Rgeg, conn=, extract_orig=F, extract_resp=F, dns=, dns_state=, ftp=, http=, http_state=, irc=, mime=, mime_state=, smtp=, smtp_state=, ssh=, ssl=, syslog=]') 1311220077.462050 /home/gilbert/Code/bro/policy/frameworks/notice/weird.bro:310 function called: Weird::report_weird(t = '1311220077.46205', name = 'irc_line_too_short', id = '212.110.251.3:52895 > 64.18.128.86:6667', have_conn = 'T', addl = '', action = 'WEIRD_FILE', no_log = 'F') 1311220077.462050 /home/gilbert/Code/bro/policy/frameworks/logging/base.bro:188 function called: Log::write(id = 'WEIRD', columns = '[ts=1311220077.46205, uid=Y6rpTW6Rgeg, id=[orig_h=212.110.251.3, orig_p=52895/tcp, resp_h=64.18.128.86, resp_p=6667/tcp], msg=irc_line_too_short, addl=, notice=F]') 1311220077.462050 /home/gilbert/Code/bro/policy/frameworks/logging/base.bro:188 Builtin Function called: Log::__write(id = 'WEIRD', columns = '[ts=1311220077.46205, uid=Y6rpTW6Rgeg, id=[orig_h=212.110.251.3, orig_p=52895/tcp, resp_h=64.18.128.86, resp_p=6667/tcp], msg=irc_line_too_short, addl=, notice=F]') 1311220077.462050 /home/gilbert/Code/bro/policy/frameworks/logging/base.bro:188 Function return: T 1311220077.462050 /home/gilbert/Code/bro/policy/frameworks/logging/base.bro:188 Function return: T 1311220077.462050 /home/gilbert/Code/bro/build/src/event.bif.bro:611 event called: irc_nick_message(c = '[id=[orig_h=212.110.251.3, orig_p=52895/tcp, resp_h=64.18.128.86, resp_p=6667/tcp], orig=[size=38, state=3, num_pkts=, num_bytes_ip=], resp=[size=0, state=0, num_pkts=, num_bytes_ip=], start_time=1311220077.46205, duration=0.0, service={ }, addl=, hot=0, history=D, uid=Y6rpTW6Rgeg, conn=, extract_orig=F, extract_resp=F, dns=, dns_state=, ftp=, http=, http_state=, irc=, mime=, mime_state=, smtp=, smtp_state=, ssh=, ssl=, syslog=]', who = 'T', newnick = '', vararg0 = 'Trance') }}} Also, gdb stack trace: {{{ (gdb) bt #0 0x081ae57f in Unref (o=0x1b9) at /home/gilbert/Code/bro- baseline/src/Obj.h:215 #1 0x0827fc24 in Frame::SetElement (this=0xac64360, n=3, v=0xac60da0) at /home/gilbert/Code/bro-baseline/src/Frame.h:25 #2 0x08288796 in BroFunc::Call (this=0x920b258, args=0xac60ec8, parent=0x0) at /home/gilbert/Code/bro-baseline/src/Func.cc:304 #3 0x08251d33 in EventHandler::Call (this=0x91363c0, vl=0xac60ec8, no_remote=false) at /home/gilbert/Code/bro-baseline/src/EventHandler.cc:73 #4 0x081fd6c9 in Event::Dispatch (this=0xac63e08, no_remote=false) at /home/gilbert/Code/bro-baseline/src/Event.h:46 #5 0x08251594 in EventMgr::Dispatch (this=0x8471da0) at /home/gilbert/Code/bro-baseline/src/Event.cc:107 #6 0x082515ef in EventMgr::Drain (this=0x8471da0) at /home/gilbert/Code /bro-baseline/src/Event.cc:119 #7 0x082e16f1 in net_packet_dispatch (t=1311220077.46205, hdr=0x9a15c58, pkt=0x9a16148 "", hdr_size=14, src_ps=0x9a15c20, pkt_elem=0x0) at /home/gilbert/Code/bro-baseline/src/Net.cc:354 #8 0x082e1912 in net_packet_arrival (t=1311220077.46205, hdr=0x9a15c58, pkt=0x9a16148 "", hdr_size=14, src_ps=0x9a15c20) at /home/gilbert/Code/bro-baseline/src/Net.cc:416 #9 0x082f2433 in PktSrc::Process (this=0x9a15c20) at /home/gilbert/Code /bro-baseline/src/PktSrc.cc:275 #10 0x082e1a33 in net_run () at /home/gilbert/Code/bro- baseline/src/Net.cc:446 #11 0x081fcf5e in main (argc=8, argv=0xbf90af14) at /home/gilbert/Code /bro-baseline/src/main.cc:997 }}} This crash was triggered by replaying a trace through a local TAP device; Bro was listening on the TAP device and processing the replayed packets. The address passed to Unref() appears to be invalid. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jul 20 21:28:30 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 21 Jul 2011 04:28:30 -0000 Subject: [Bro-Dev] #505: Invalid Unref crash In-Reply-To: <048.a3ead8ea8da3e7844a25274cbfce1df6@tracker.bro-ids.org> References: <048.a3ead8ea8da3e7844a25274cbfce1df6@tracker.bro-ids.org> Message-ID: <063.e6500f4403a473ca5ffe7318f3e0a66c@tracker.bro-ids.org> #505: Invalid Unref crash ----------------------+------------------------ Reporter: gclark | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by seth): > This crash was triggered by replaying a trace through a local TAP device; > Bro was listening on the TAP device and processing the replayed packets. Those may be null encapsulated packets which may not be supported or supported correctly. -- Ticket URL: Bro Tracker Bro Issue Tracker From hlin33 at illinois.edu Wed Jul 20 21:41:12 2011 From: hlin33 at illinois.edu (Hui Lin) Date: Wed, 20 Jul 2011 21:41:12 -0700 Subject: [Bro-Dev] Hui Lin_Binpac: handle incremental input for flowunit Message-ID: HI, When I define flowunit data for flow, I am bothered by the not support incremental input. I have searched this error and found a mail from Seth. It seems that flowunit data has to be clearly set the &length. My situation is something like follows type XXX_Request = record { header: Header; data: case header.function_code of { x -> request_type1: Type1; x -> request_type2: Type2; .............. }; } &length = ....; XXX_Request is the flowunit data. My problem is that after header is parsed, I still don't know the length of the whole XXX_Request data unit. The function_code from the header will define what data type follows the header and different data type have different length. So I am wondering what I can do in this situation to set the proper value of length of the XXX_Request. Or flowunit is not supporting this situation, I have to use the datagram? Best, Hui -- Hui Lin Research Assistant DEPEND Research Group, ECE Department University of Illinois at Urbana-Champaign hlin33 at illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20110720/0b5814c9/attachment.html From bro at tracker.bro-ids.org Wed Jul 20 21:46:55 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 21 Jul 2011 04:46:55 -0000 Subject: [Bro-Dev] #506: loggable functions Message-ID: <046.fd94fc1efd6f020fa77217ca380cf485@tracker.bro-ids.org> #506: loggable functions ---------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: preview | ---------------------+-------------------- Functions should be &log-able. The ascii writer would turn the function into a single line representation and other output plugins would handle it however they needed to. -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Wed Jul 20 21:56:35 2011 From: seth at icir.org (Seth Hall) Date: Thu, 21 Jul 2011 00:56:35 -0400 Subject: [Bro-Dev] Hui Lin_Binpac: handle incremental input for flowunit In-Reply-To: References: Message-ID: <0FC0CB94-BB73-48E4-8B47-906CFC747B47@icir.org> On Jul 21, 2011, at 12:41 AM, Hui Lin wrote: > XXX_Request is the flowunit data. My problem is that after header is parsed, I still don't know the length of the whole XXX_Request data unit. Presumably this is for the DNP3 protocol? There isn't any framing around the request with the length? Are they framing it to the packet perhaps? (I'm running into sort of similar trouble with the SSL analyzer and binpac doesn't let you frame to the packets). I don't think you can use datagram if the protocol you're working on is TCP since the packets could be fragmented or out of order. I'd love to be proven wrong though.... .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From hlin33 at illinois.edu Wed Jul 20 22:18:34 2011 From: hlin33 at illinois.edu (Hui Lin) Date: Wed, 20 Jul 2011 22:18:34 -0700 Subject: [Bro-Dev] Hui Lin_Binpac: handle incremental input for flowunit In-Reply-To: <0FC0CB94-BB73-48E4-8B47-906CFC747B47@icir.org> References: <0FC0CB94-BB73-48E4-8B47-906CFC747B47@icir.org> Message-ID: It is the DNP3 protocol. I kind of refer to the modbus protocol that shared from Dina. In modbus, the header contain the length field, so it is straightforward to set the length of the flowunit. But DNP3 does not contain such field to directly indicate the length of the application level fragment. The structure is kind of more complex. Some request will contain addition headers and objects, so you can only know the length of the whole fragment when you parse them all. Or at least parse this additional header (but there is also no length field in this additional header). I don't quite follow what do you mean "framing it to the packet". Any further suggestion? On Wed, Jul 20, 2011 at 9:56 PM, Seth Hall wrote: > > On Jul 21, 2011, at 12:41 AM, Hui Lin wrote: > > > XXX_Request is the flowunit data. My problem is that after header is > parsed, I still don't know the length of the whole XXX_Request data unit. > > Presumably this is for the DNP3 protocol? There isn't any framing around > the request with the length? Are they framing it to the packet perhaps? > (I'm running into sort of similar trouble with the SSL analyzer and binpac > doesn't let you frame to the packets). > > I don't think you can use datagram if the protocol you're working on is TCP > since the packets could be fragmented or out of order. I'd love to be > proven wrong though.... > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro-ids.org/ > > -- Hui Lin Research Assistant DEPEND Research Group, ECE Department University of Illinois at Urbana-Champaign hlin33 at illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20110720/3ff621f4/attachment.html From gc355804 at ohio.edu Wed Jul 20 22:25:11 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Thu, 21 Jul 2011 01:25:11 -0400 Subject: [Bro-Dev] #505: Invalid Unref crash In-Reply-To: <063.e6500f4403a473ca5ffe7318f3e0a66c@tracker.bro-ids.org> References: <048.a3ead8ea8da3e7844a25274cbfce1df6@tracker.bro-ids.org> <063.e6500f4403a473ca5ffe7318f3e0a66c@tracker.bro-ids.org> Message-ID: <4E27B837.1090802@ohio.edu> Hmm; good point. I found it odd that the new logging framework code doesn't crash given the same trace under the same conditions, though. Also, since the crash seems to be in the core, I'd be a little surprised if it was the threading / logging code in there that was doing it. I *did* just notice that the script path in that trace doesn't match the path to the bro executable; if the problem appears on a real interface as well, will re-run and generate a correct trace :) --Gilbert On 7/21/2011 12:28 AM, Bro Tracker wrote: > #505: Invalid Unref crash > ----------------------+------------------------ > Reporter: gclark | Owner: > Type: Problem | Status: new > Priority: Normal | Milestone: > Component: Bro | Version: git/master > Resolution: | Keywords: > ----------------------+------------------------ > > Comment (by seth): > > > This crash was triggered by replaying a trace through a local TAP > device; > > Bro was listening on the TAP device and processing the replayed packets. > > Those may be null encapsulated packets which may not be supported or > supported correctly. > From bro at tracker.bro-ids.org Wed Jul 20 22:25:52 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 21 Jul 2011 05:25:52 -0000 Subject: [Bro-Dev] #505: Invalid Unref crash In-Reply-To: <048.a3ead8ea8da3e7844a25274cbfce1df6@tracker.bro-ids.org> References: <048.a3ead8ea8da3e7844a25274cbfce1df6@tracker.bro-ids.org> Message-ID: <063.045e5e158a8d4ca708e5b963995f18c7@tracker.bro-ids.org> #505: Invalid Unref crash ----------------------+------------------------ Reporter: gclark | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by gclark): Hmm; good point. I found it odd that the new logging framework code doesn't crash given the same trace under the same conditions, though. Also, since the crash seems to be in the core, I'd be a little surprised if it was the threading / logging code in there that was doing it. I *did* just notice that the script path in that trace doesn't match the path to the bro executable; if the problem appears on a real interface as well, will re-run and generate a correct trace :) --Gilbert On 7/21/2011 12:28 AM, Bro Tracker wrote: >#505: Invalid Unref crash >----------------------+------------------------ > Reporter: gclark | Owner: > Type: Problem | Status: new > Priority: Normal | Milestone: > Component: Bro | Version: git/master >Resolution: | Keywords: >----------------------+------------------------ > >Comment (by seth): > > > This crash was triggered by replaying a trace through a local TAP > device; > > Bro was listening on the TAP device and processing the replayed packets. > > Those may be null encapsulated packets which may not be supported or > supported correctly. > -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jul 20 22:44:05 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 21 Jul 2011 05:44:05 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: AccopAdatty Message-ID: <20110721054403.8649136A391@taffy.ICSI.Berkeley.EDU> New user registration for user AccopAdatty -- Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jul 21 04:34:49 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 21 Jul 2011 11:34:49 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: JigMaster Message-ID: <20110721113449.09F3436A377@taffy.ICSI.Berkeley.EDU> New user registration for user JigMaster -- Bro Tracker Bro Issue Tracker From seth at icir.org Thu Jul 21 04:51:48 2011 From: seth at icir.org (Seth Hall) Date: Thu, 21 Jul 2011 07:51:48 -0400 Subject: [Bro-Dev] Hui Lin_Binpac: handle incremental input for flowunit In-Reply-To: References: <0FC0CB94-BB73-48E4-8B47-906CFC747B47@icir.org> Message-ID: <0E5592AE-B130-45F5-9A38-A0BE8B3B43C7@icir.org> On Jul 21, 2011, at 1:18 AM, Hui Lin wrote: > The structure is kind of more complex. Some request will contain addition headers and objects, so you can only know the length of the whole fragment when you parse them all. Or at least parse this additional header (but there is also no length field in this additional header). I don't quite follow what do you mean "framing it to the packet". Any further suggestion? I just mean using the packet length as the length of the request or response. If that's how the protocol works and if binpac supported it (a lot of "if"s), that would provide you the way to give a length to the top parse unit to avoid the incremental parsing error. Maybe someone else has a suggestion? I'm unfortunately out of ideas. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From gregor at icir.org Thu Jul 21 11:40:53 2011 From: gregor at icir.org (Gregor Maier) Date: Thu, 21 Jul 2011 11:40:53 -0700 Subject: [Bro-Dev] Bro workshop slides and exercises on the website Message-ID: <4E2872B5.1040907@icir.org> Hi, I think we should put the Bro workshop slides and exercises on the homepage. The ones from the 2007 workshop are linked from the old wiki but the link is now broken. I couldn't find the slides or exercises for the 2009 workshop. IMHO those slides might help users getting a good start with Bro. cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From slagell at ncsa.illinois.edu Thu Jul 21 11:47:20 2011 From: slagell at ncsa.illinois.edu (Adam J. Slagell) Date: Thu, 21 Jul 2011 13:47:20 -0500 Subject: [Bro-Dev] Bro workshop slides and exercises on the website In-Reply-To: <4E2872B5.1040907@icir.org> References: <4E2872B5.1040907@icir.org> Message-ID: <2BF287E2-7919-4E01-B55B-CA24C6BC43ED@ncsa.illinois.edu> That is a good point. All of us working with Bro at NCSA have gone through those materials. On Jul 21, 2011, at 1:40 PM, Gregor Maier wrote: > Hi, > > I think we should put the Bro workshop slides and exercises on the > homepage. The ones from the 2007 workshop are linked from the old wiki > but the link is now broken. I couldn't find the slides or exercises for > the 2009 workshop. IMHO those slides might help users getting a good > start with Bro. > > cu > Gregor > -- > Gregor Maier > > Int. Computer Science Institute (ICSI) > 1947 Center St., Ste. 600 > Berkeley, CA 94704, USA > http://www.icir.org/gregor/ > _______________________________________________ > bro-dev mailing list > bro-dev at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > ------ Adam J. Slagell, CISSP Sr. Security Engineer National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info 217.244.8965 From bro at tracker.bro-ids.org Thu Jul 21 13:00:09 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 21 Jul 2011 20:00:09 -0000 Subject: [Bro-Dev] #458: Debian Packaging via CPack In-Reply-To: <048.9b1834486ac88d67b57ae6dc4688573b@tracker.bro-ids.org> References: <048.9b1834486ac88d67b57ae6dc4688573b@tracker.bro-ids.org> Message-ID: <063.c4fb0d68118dc70a3fb221df58d9cbc8@tracker.bro-ids.org> #458: Debian Packaging via CPack -----------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Comment (by robin): In [3d2479f4751c754ad4911eeff9f32b54753ac5ae/broctl]: {{{ #!CommitTicketReference repository="broctl" revision="3d2479f4751c754ad4911eeff9f32b54753ac5ae" Merge remote-tracking branch 'remotes/origin/topic/policy-scripts-new' into topic/robin/plugins * remotes/origin/topic/policy-scripts-new: BroCtl top command finally completely works on MacOS X. Fixed another problem to avoid scientific notation conversion in byte conversion. Fixing broctl's top output on some operating systems. Crash diagnostics ended up with trash from gdb sometimes. The SendMail option is now where the sendmail binary is configured. More updates for BroControl Accidently wasn't loading the cluster framework. Fixed loading of analysis-groups. Checkpoint for broctl port to new policy scripts. Updating submodule(s). Standardize shell script hashbang on install. Fix binary package broctl-config.sh symlink installation regression Changes to allow DEB packaging via CPack, addresses #458 Fixed a problem with the "update" command. Updating submodule(s). Update CHANGES. Updating submodule(s). This is *completely* untested, and just in preparation for the later merge of both branches into master. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jul 21 13:22:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 21 Jul 2011 20:22:13 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: Reiciestaries Message-ID: <20110721202212.5D8B8369FE2@taffy.ICSI.Berkeley.EDU> New user registration for user Reiciestaries -- Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jul 21 13:39:33 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 21 Jul 2011 20:39:33 -0000 Subject: [Bro-Dev] #507: topic/jsiwek/pybroccoli-float-repr Message-ID: <048.dc0a250607db12c19285690426bd3061@tracker.bro-ids.org> #507: topic/jsiwek/pybroccoli-float-repr ---------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ This branch is in `broccoli-python` and `bro` repos and should make the pybroccoli unit test work across more versions of python (Python 2.7's `repr()` for floats may display a shorter string). Should also be less sensitive to the rounding of floats derived from current time, which sometimes ends up bypassing the regex in the timestamp diff canonifier. -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Thu Jul 21 14:01:06 2011 From: seth at icir.org (Seth Hall) Date: Thu, 21 Jul 2011 17:01:06 -0400 Subject: [Bro-Dev] [Bro Tracker] New user registration: Reiciestaries In-Reply-To: <20110721202212.5D8B8369FE2@taffy.ICSI.Berkeley.EDU> References: <20110721202212.5D8B8369FE2@taffy.ICSI.Berkeley.EDU> Message-ID: <3B8C58E4-D501-4808-83C0-B138CDF003B7@icir.org> On Jul 21, 2011, at 4:22 PM, Bro Tracker wrote: > New user registration for user Reiciestaries Ugh, no more self account creation. That's way too annoying. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From slagell at ncsa.illinois.edu Thu Jul 21 14:05:39 2011 From: slagell at ncsa.illinois.edu (Adam J. Slagell) Date: Thu, 21 Jul 2011 16:05:39 -0500 Subject: [Bro-Dev] Hui Lin_Binpac: handle incremental input for flowunit In-Reply-To: <0FC0CB94-BB73-48E4-8B47-906CFC747B47@icir.org> References: <0FC0CB94-BB73-48E4-8B47-906CFC747B47@icir.org> Message-ID: <361A7028-9D99-4504-9445-A5CD23D6FFED@ncsa.illinois.edu> On Jul 20, 2011, at 11:56 PM, Seth Hall wrote: > I don't think you can use datagram if the protocol you're working on is TCP since the packets could be fragmented or out of order. I'd love to be proven wrong though.... Obviously, modbus-tcp was on tcp, but is DNP3? ------ Adam J. Slagell, CISSP Sr. Security Engineer National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info 217.244.8965 From bro at tracker.bro-ids.org Thu Jul 21 15:20:24 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 21 Jul 2011 22:20:24 -0000 Subject: [Bro-Dev] #507: topic/jsiwek/pybroccoli-float-repr In-Reply-To: <048.dc0a250607db12c19285690426bd3061@tracker.bro-ids.org> References: <048.dc0a250607db12c19285690426bd3061@tracker.bro-ids.org> Message-ID: <063.829cd43b09828d393d3b4053e54f65bf@tracker.bro-ids.org> #507: topic/jsiwek/pybroccoli-float-repr ----------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by robin): On Thu, Jul 21, 2011 at 20:39 -0000, you wrote: > This branch is in `broccoli-python` and `bro` repos and should make the > pybroccoli unit test work across more versions of python (Python 2.7's > `repr()` for floats may display a shorter string). Should also be less > sensitive to the rounding of floats derived from current time, which > sometimes ends up bypassing the regex in the timestamp diff canonifier. Haven't looked at the change yet, but there's a related other isssue: the fact that the ASCII logging currently outputs less than 6 digits after the comma for timestamps that end in zeros, can make the output hard to read because subsequent columns aren't aligned. We should change that so that it always print the same number of digits. Robin -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jul 21 15:29:24 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 21 Jul 2011 22:29:24 -0000 Subject: [Bro-Dev] #497: bro -Z generates a file with "internal error" in its name In-Reply-To: <046.949382cf5955c7a7219675b37f0cad41@tracker.bro-ids.org> References: <046.949382cf5955c7a7219675b37f0cad41@tracker.bro-ids.org> Message-ID: <061.5e9a15434764a7689cfbece835d0c916@tracker.bro-ids.org> #497: bro -Z generates a file with "internal error" in its name ----------------------+-------------------- Reporter: vern | Owner: jsiwek Type: Problem | Status: closed Priority: Low | Milestone: Component: Bro | Version: Resolution: fixed | Keywords: ----------------------+-------------------- Comment (by robin): In [eb1e76600c90a48435b8c1263c9073028b406064/bro]: {{{ #!CommitTicketReference repository="bro" revision="eb1e76600c90a48435b8c1263c9073028b406064" Merge remote-tracking branch 'origin/topic/script-load-changes' * origin/topic/script-load-changes: Fix reST file name associated w/ stdin when in doc mode (closes #497) Update @prefixes test. Rewrite a test using btest's TEST-START-FILE directive Fix @unload'd files from generating bro_script_loaded event. Renaming a test better. Reimplementation of the @prefixes statement. Fix accidental overwrite of BROPATH copy. Make @load statements recognize relative paths. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jul 21 15:29:35 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 21 Jul 2011 22:29:35 -0000 Subject: [Bro-Dev] #504: topic/script-load-changes In-Reply-To: <048.569587bed68aec0d9cf35b76f39daad8@tracker.bro-ids.org> References: <048.569587bed68aec0d9cf35b76f39daad8@tracker.bro-ids.org> Message-ID: <063.3ae18034076d112da4058fd134021d50@tracker.bro-ids.org> #504: topic/script-load-changes -----------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jul 21 15:34:02 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 21 Jul 2011 22:34:02 -0000 Subject: [Bro-Dev] #506: loggable functions In-Reply-To: <046.fd94fc1efd6f020fa77217ca380cf485@tracker.bro-ids.org> References: <046.fd94fc1efd6f020fa77217ca380cf485@tracker.bro-ids.org> Message-ID: <061.2793c54b2602c844d873a3586cc84c79@tracker.bro-ids.org> #506: loggable functions ----------------------+--------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: preview ----------------------+--------------------- Comment (by robin): On Thu, Jul 21, 2011 at 04:46 -0000, you wrote: > #506: loggable functions > ---------------------+-------------------- > Reporter: seth | Owner: > Type: Problem | Status: new > Priority: Normal | Milestone: Bro1.6 > Component: Bro | Version: > Keywords: preview | > ---------------------+-------------------- > Functions should be &log-able. The ascii writer would turn the function > into a single line representation and other output plugins would handle it > however they needed to. > Do you mean the full function body? I have a guess where you want to that for but I'll ask nevertheless: what do you want to use that for? :) We can do that, but I would prefer to generally convert to string and then pass as string to all types of writers. Fine? Robin -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jul 21 17:13:15 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 22 Jul 2011 00:13:15 -0000 Subject: [Bro-Dev] #506: loggable functions In-Reply-To: <046.fd94fc1efd6f020fa77217ca380cf485@tracker.bro-ids.org> References: <046.fd94fc1efd6f020fa77217ca380cf485@tracker.bro-ids.org> Message-ID: <061.f492dbdc7a37c82e123e596e19939327@tracker.bro-ids.org> #506: loggable functions ----------------------+--------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: preview ----------------------+--------------------- Comment (by seth): > Do you mean the full function body? I have a guess where you want to > that for but I'll ask nevertheless: what do you want to use that for? :) Yes you do. The notice framework auditing log (notice_policy.log) > We can do that, but I would prefer to generally convert to string and > then pass as string to all types of writers. Fine? That seems like the best solution to me too. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jul 21 17:25:58 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 22 Jul 2011 00:25:58 -0000 Subject: [Bro-Dev] #506: loggable functions In-Reply-To: <046.fd94fc1efd6f020fa77217ca380cf485@tracker.bro-ids.org> References: <046.fd94fc1efd6f020fa77217ca380cf485@tracker.bro-ids.org> Message-ID: <061.3c78785f911d9d1f5582a98d2d1d9bef@tracker.bro-ids.org> #506: loggable functions ----------------------+---------------------- Reporter: seth | Owner: robin Type: Problem | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: preview ----------------------+---------------------- Changes (by robin): * owner: => robin * status: new => assigned -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jul 21 18:25:09 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 22 Jul 2011 01:25:09 -0000 Subject: [Bro-Dev] #473: Remove or redirect "bro_logger" messages. In-Reply-To: <046.ac0045106dd553365c25462db1b26a3d@tracker.bro-ids.org> References: <046.ac0045106dd553365c25462db1b26a3d@tracker.bro-ids.org> Message-ID: <061.09d31f6e426138921350bdb48451d64c@tracker.bro-ids.org> #473: Remove or redirect "bro_logger" messages. ----------------------+------------------------ Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by seth): I lost track, has this been done? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jul 21 18:32:19 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 22 Jul 2011 01:32:19 -0000 Subject: [Bro-Dev] #508: ASCII Log Writer precision of time values varies Message-ID: <048.2c96fb316543c967f2184acb450ad82b@tracker.bro-ids.org> #508: ASCII Log Writer precision of time values varies -----------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | -----------------------------+------------------------ It would be nicer if, by default, time values were written with a constant number of digits after the decimal. Should we: 1) Hardcode a constant 6 digit precision 2) Make the precision a configurable `&redef` option of the writer, defaulting to 6 3) Make a configurable `&redef` option of the writer that can be an arbitrary function that takes a time value and returns the formatted timestamp string for it, defaulting to a function that returns the 6 digit precision (e.g. could also provide a function that returns a human readable timestamp string) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jul 21 19:33:46 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 22 Jul 2011 02:33:46 -0000 Subject: [Bro-Dev] #508: ASCII Log Writer precision of time values varies In-Reply-To: <048.2c96fb316543c967f2184acb450ad82b@tracker.bro-ids.org> References: <048.2c96fb316543c967f2184acb450ad82b@tracker.bro-ids.org> Message-ID: <063.0ead451966f4f9228609b6207ed4d659@tracker.bro-ids.org> #508: ASCII Log Writer precision of time values varies ------------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ------------------------------+------------------------ Comment (by seth): > 1) Hardcode a constant 6 digit precision Personally I'm in favor of this. I have a hard time someone would care too much about it. At some point in the future I have to think that we may have some more abstracted method to modify field contents through filters or something anyway. -- Ticket URL: Bro Tracker Bro Issue Tracker From gc355804 at ohio.edu Thu Jul 21 19:39:25 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Thu, 21 Jul 2011 22:39:25 -0400 Subject: [Bro-Dev] #508: ASCII Log Writer precision of time values varies In-Reply-To: <048.2c96fb316543c967f2184acb450ad82b@tracker.bro-ids.org> References: <048.2c96fb316543c967f2184acb450ad82b@tracker.bro-ids.org> Message-ID: <4E28E2DD.90507@ohio.edu> 3) sounds like it could lead to thread-safety issues with the new logging stuff. Rather than a user-defined function, though, what about allowing the user to specify a date string (with additional UNIX second / fractional second options)? --Gilbert On 7/21/2011 9:32 PM, Bro Tracker wrote: > #508: ASCII Log Writer precision of time values varies > -----------------------------+------------------------ > Reporter: jsiwek | Owner: jsiwek > Type: Feature Request | Status: new > Priority: Normal | Milestone: Bro1.6 > Component: Bro | Version: git/master > Keywords: | > -----------------------------+------------------------ > It would be nicer if, by default, time values were written with a constant > number of digits after the decimal. Should we: > > 1) Hardcode a constant 6 digit precision > 2) Make the precision a configurable `&redef` option of the writer, > defaulting to 6 > 3) Make a configurable `&redef` option of the writer that can be an > arbitrary function that takes a time value and returns the formatted > timestamp string for it, defaulting to a function that returns the 6 digit > precision (e.g. could also provide a function that returns a human > readable timestamp string) > From robin at icir.org Thu Jul 21 20:49:16 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 21 Jul 2011 20:49:16 -0700 Subject: [Bro-Dev] #508: ASCII Log Writer precision of time values varies In-Reply-To: <4E28E2DD.90507@ohio.edu> References: <048.2c96fb316543c967f2184acb450ad82b@tracker.bro-ids.org> <4E28E2DD.90507@ohio.edu> Message-ID: <20110722034916.GA110@icir.org> On Thu, Jul 21, 2011 at 22:39 -0400, you wrote: > Rather than a user-defined function, though, what about allowing the > user to specify a date string (with additional UNIX second / fractional > second options)? I generally like this but strftime doesn't give us what we need and I'm not sure rolling our own is worth it. So harcoding 6 digits would be fine with me too. (At some point we want higher resolution, but there'll be many more places to touch for that ...) Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Thu Jul 21 20:51:15 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 22 Jul 2011 03:51:15 -0000 Subject: [Bro-Dev] #473: Remove or redirect "bro_logger" messages. In-Reply-To: <046.ac0045106dd553365c25462db1b26a3d@tracker.bro-ids.org> References: <046.ac0045106dd553365c25462db1b26a3d@tracker.bro-ids.org> Message-ID: <061.01ece63ca4248e75d13f0ddfe4322089@tracker.bro-ids.org> #473: Remove or redirect "bro_logger" messages. ----------------------+------------------------ Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by robin): On Fri, Jul 22, 2011 at 01:25 -0000, you wrote: > I lost track, has this been done? Yes, should be going through the scripting layer now. Closing ticket. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Thu Jul 21 21:00:29 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 21 Jul 2011 21:00:29 -0700 Subject: [Bro-Dev] Bro workshop slides and exercises on the website In-Reply-To: <2BF287E2-7919-4E01-B55B-CA24C6BC43ED@ncsa.illinois.edu> References: <4E2872B5.1040907@icir.org> <2BF287E2-7919-4E01-B55B-CA24C6BC43ED@ncsa.illinois.edu> Message-ID: <20110722040029.GA32586@icir.org> On Thu, Jul 21, 2011 at 13:47 -0500, you wrote: > That is a good point. All of us working with Bro at NCSA have gone through those materials. I'll see that get them back online and linked. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From hlin33 at illinois.edu Thu Jul 21 21:01:51 2011 From: hlin33 at illinois.edu (Hui Lin) Date: Thu, 21 Jul 2011 21:01:51 -0700 Subject: [Bro-Dev] Hui Lin_Binpac: handle incremental input for flowunit In-Reply-To: <361A7028-9D99-4504-9445-A5CD23D6FFED@ncsa.illinois.edu> References: <0FC0CB94-BB73-48E4-8B47-906CFC747B47@icir.org> <361A7028-9D99-4504-9445-A5CD23D6FFED@ncsa.illinois.edu> Message-ID: DNP3 protocol do include some support for what it calls Pseudo Transport Layer. And it also mentions that some company would like to see DNP3 can be put on TCP/IP. So I would like to try use flowunit to implement DNP3 protocol. On Thu, Jul 21, 2011 at 2:05 PM, Adam J. Slagell wrote: > > On Jul 20, 2011, at 11:56 PM, Seth Hall wrote: > > > I don't think you can use datagram if the protocol you're working on is > TCP since the packets could be fragmented or out of order. I'd love to be > proven wrong though.... > > Obviously, modbus-tcp was on tcp, but is DNP3? > > ------ > Adam J. Slagell, CISSP > Sr. Security Engineer > National Center for Supercomputing Applications > University of Illinois at Urbana-Champaign > www.slagell.info > 217.244.8965 > > > > > > > -- Hui Lin Research Assistant DEPEND Research Group, ECE Department University of Illinois at Urbana-Champaign hlin33 at illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20110721/df92b51e/attachment.html From robin at icir.org Thu Jul 21 21:20:53 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 21 Jul 2011 21:20:53 -0700 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <1310914521-sup-1714@samurai.icir.org> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> <20110707215133.GF77684@icir.org> <4E163B71.6050804@ohio.edu> <20110708132228.GA84600@icir.org> <1310914521-sup-1714@samurai.icir.org> Message-ID: <20110722042053.GB32586@icir.org> On Sun, Jul 17, 2011 at 08:32 -0700, you wrote: > threads, tasks/futures, smart pointers, SFINAE helpers like > boost::enable_if, RNGs, etc. Overall, I find myself needing less time to > write more code that actually does something. I believe all of that! However, my main concern is that I don't want to introduce such functionality through the backdoor by starting to use it here and there. That leads to code that's really hard to maintain because one suddenly has a mix of different styles, idioms, level of abstractions, etc. For Bro, we had this in the past already with STL containers: they slowly started to being used more and more, and now we have quite a mix all over the code. I like the STL containers and I certainly prefer to use them over the macro-based alternatives (and I do use them where I can these days), but I still don't think that having both versions in the same code base is a good thing. Containers aren't really that critical in this context as it's rather clear to see what's going on (though there are cases where we now need to convert from container type A to B). But I'm not sure that also applies to all the shiny new features in C++11 ... So, generally, I'd be fine switching to using more modern stuff but I think that needs to be a conscious decision and we need to adapt the existing code base accordingly. We also need to make sure we have the platform support everywhere we need it; don't think we are there yet (or are we?). Coming back to the original question, in my view triggering all this is not really justified just for doing threaded loggging. Longer term, it may be fine. > If we really want to use Boost (say to have a platform-independent > networking library), we could whitelist the Boost components that we > allow in Bro. That's a slippery slope. With that, everytime somebody wants to use something else (which is always tempting of course :), we'd need to argue about it and potentially find arguments for *not* using it; which will be hard I'm sure. Or somebody might just start using it and "will change that later" ... (Btw, DataSeries uses Boost. So folks using that will need that dependency already. And it's quite a bit one as I noticed when installing the Boost FreeBSD port the other day ...) Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From hlin33 at illinois.edu Thu Jul 21 21:28:46 2011 From: hlin33 at illinois.edu (Hui Lin) Date: Thu, 21 Jul 2011 21:28:46 -0700 Subject: [Bro-Dev] Hui Lin_Binpac: handle incremental input for flowunit In-Reply-To: <0E5592AE-B130-45F5-9A38-A0BE8B3B43C7@icir.org> References: <0FC0CB94-BB73-48E4-8B47-906CFC747B47@icir.org> <0E5592AE-B130-45F5-9A38-A0BE8B3B43C7@icir.org> Message-ID: Hi Seth, Here is a piece code from SSL protocol (as you mentioned that you met the similar problem in SSL protocol) type SSLRecord(is_orig: bool) = record { head0 : uint8; head1 : uint8; head2 : uint8; head3 : uint8; head4 : uint8; rec : RecordText(this, is_orig) &requires(content_type), &restofdata; } &length = length+5, &byteorder=bigendian, &let { version : int = $context.analyzer.determine_ssl_version(head0, head1, head2); content_type : int = case version of { UNKNOWN_VERSION -> 0; SSLv20 -> head2+300; default -> head0; }; length : int = case version of { UNKNOWN_VERSION -> 0; SSLv20 -> (((head0 & 0x7f) << 8) | head1) - 3; default -> (head3 << 8) | head4; }; }; Here the &length is set by its attribute length and length is then decided according to different situation. so I think this might be working in my case? As I can similarly set length to different value according to the function code and then use this to set the &length. On Thu, Jul 21, 2011 at 4:51 AM, Seth Hall wrote: > > On Jul 21, 2011, at 1:18 AM, Hui Lin wrote: > > > The structure is kind of more complex. Some request will contain addition > headers and objects, so you can only know the length of the whole fragment > when you parse them all. Or at least parse this additional header (but there > is also no length field in this additional header). I don't quite follow > what do you mean "framing it to the packet". Any further suggestion? > > I just mean using the packet length as the length of the request or > response. If that's how the protocol works and if binpac supported it (a > lot of "if"s), that would provide you the way to give a length to the top > parse unit to avoid the incremental parsing error. > > Maybe someone else has a suggestion? I'm unfortunately out of ideas. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro-ids.org/ > > -- Hui Lin Research Assistant DEPEND Research Group, ECE Department University of Illinois at Urbana-Champaign hlin33 at illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20110721/931c3e98/attachment-0001.html From bro at tracker.bro-ids.org Fri Jul 22 04:13:33 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 22 Jul 2011 11:13:33 -0000 Subject: [Bro-Dev] #509: SMTP rework Message-ID: <046.35777c113d149585390ec44632cce689@tracker.bro-ids.org> #509: SMTP rework ---------------------+------------------------ Reporter: seth | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: preview | ---------------------+------------------------ There are a couple of tasks that need done on the SMTP analyzer before the preview. - Stop doing message header parsing with the smtp_data event and use the mime_* events (they deal with header wrap around appropriately and the various message encodings). - Create a new logging stream named something like SMTP::ENTITIES to log file information. It would have the same information as the current mim.log. -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Fri Jul 22 04:17:11 2011 From: seth at icir.org (Seth Hall) Date: Fri, 22 Jul 2011 07:17:11 -0400 Subject: [Bro-Dev] Hui Lin_Binpac: handle incremental input for flowunit In-Reply-To: References: <0FC0CB94-BB73-48E4-8B47-906CFC747B47@icir.org> <0E5592AE-B130-45F5-9A38-A0BE8B3B43C7@icir.org> Message-ID: On Jul 22, 2011, at 12:28 AM, Hui Lin wrote: > Here the &length is set by its attribute length and length is then decided according to different situation. so I think this might be working in my case? As I can similarly set length to different value according to the function code and then use this to set the &length. Yes, you should be able to do something similar to that. I can't claim to have come up with that technique either. Whoever wrote the previous binpac SSL analyzer did it, but we are currently having some trouble with that when DPD is enabled. We're trying to get it fixed for the preview though. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Fri Jul 22 08:52:50 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 22 Jul 2011 15:52:50 -0000 Subject: [Bro-Dev] #508: ASCII Log Writer precision of time values varies In-Reply-To: <048.2c96fb316543c967f2184acb450ad82b@tracker.bro-ids.org> References: <048.2c96fb316543c967f2184acb450ad82b@tracker.bro-ids.org> Message-ID: <063.b59cbe13f57b17096e89c57816ba570e@tracker.bro-ids.org> #508: ASCII Log Writer precision of time values varies ------------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Feature Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: fixed | Keywords: ------------------------------+------------------------ Changes (by jsiwek): * status: new => closed * resolution: => fixed Comment: In [d30a335dba88080ec67dc5e612427af7cce8bd39/bro]: {{{ #!CommitTicketReference repository="bro" revision="d30a335dba88080ec67dc5e612427af7cce8bd39" LogWriterAscii now prints time values w/ constant 6 digit precision. closes #508 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Jul 22 08:55:57 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 22 Jul 2011 15:55:57 -0000 Subject: [Bro-Dev] #510: Upgrade How-To Message-ID: <047.e793bc8325e1f7ba35cd0d2c4f23760e@tracker.bro-ids.org> #510: Upgrade How-To ------------------------+--------------------- Reporter: robin | Type: Problem Status: new | Priority: Normal Milestone: Bro1.6 | Component: Bro Version: git/master | Keywords: preview ------------------------+--------------------- We need an Upgrade How-To summarizing the most important pieces for upgrading from 1.5 to 2.0. This is something that needs to come with the preview already I think. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Fri Jul 22 09:26:42 2011 From: robin at icir.org (Robin Sommer) Date: Fri, 22 Jul 2011 09:26:42 -0700 Subject: [Bro-Dev] Quickstart guide feedback Message-ID: <20110722162642.GL28458@icir.org> Taking a closer look at the quickstart guide was still on my todo list. It's very nice! but some thoughts about structure: - I'd switch the order of "Reading from a Trace" vs "Live Traffic" The former seems to be the more "natural" deployment model for a new user. - How about moving the "Capturing As Unprivileged User" section out of the main flow, perhaps into an appendix or even into the FAQ and then with a link from the Quickstart guide to there. It feels a bit distracting where it's right now; but it's actually also something that's quite relevant outside of the concept of "quickstarting". - The Bro Control part: - I think the link between running from the command-line and using broctl doesn't become quite clear. A bit more context upfront in the broctl section on what's it's doing and why/when one wants to use it would be helpful. - I'm also wondering if broctl should be discussed first, and the command-line version afterwards and framed as "here's the bare bones version if you want more control". - The use should also edit networks.cfg and broctl.cfg right away (for the latter at least point out how to change the recipient address for mails; that's probably the most common change). - At the end, not only mention the help command but also link to the broctl README. - The checksum discussion: is that another part for the FAQ, with a link from the Quickstart guide to there? - The configure/customize part: per above, I think this should also start with doing customizations via BroControl: where's the local policy I can edit; an example of what I put there (local_nets isn't a good one here because broctl already takes care of that via networks.cfg); and what do I do to put the change into place ("broctl check"; "install"; and "restart"). - $PREFIX/etc/analysis.dat isn't meant to be user-visible. Taking these together, what I would suggest I think is to actually have one section just on BroControl, with the corresponding parts taken out of the current running/configuraion sections; and then a separate section on just command-line usage. Does that make sense? Couple further thoughts: - Don't remember whether we talked abot this already, but navigation links between the sections would be helpul. - For the preview, we should add a note that binary packages won't be available before the final release. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Fri Jul 22 09:37:55 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 22 Jul 2011 16:37:55 -0000 Subject: [Bro-Dev] #306: Write a new user manual In-Reply-To: <046.b5f6eda94a2759af201753eaf30711c5@tracker.bro-ids.org> References: <046.b5f6eda94a2759af201753eaf30711c5@tracker.bro-ids.org> Message-ID: <061.e5864fd875889c90501c08ced3c27b60@tracker.bro-ids.org> #306: Write a new user manual ---------------------+-------------------- Reporter: seth | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ---------------------+-------------------- Changes (by robin): * milestone: Bro1.7 => Bro1.6 Old description: New description: I'm changing the milestone on this. I keep thinking that we should have something more comprehensive than the quick-start guide already for the 2.0 release (not preview). The lack of more documentation is the number one hurdle people run into, and all the changes in 2.0 don't make that better. So I'm thinking we should aim at least for some initial material explaining usage, configuration, and writing own script code in more detail. For 2.0, it doesn't need to be the fully comprehensive guide that we are planing longer-term; we can add more material later. -- -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Fri Jul 22 16:31:59 2011 From: robin at icir.org (Robin Sommer) Date: Fri, 22 Jul 2011 16:31:59 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] fastpath: Change policy.misc.loaded-scripts tests. (cd5c564) In-Reply-To: <201107222303.p6MN3ubr013067@bro-ids.icir.org> References: <201107222303.p6MN3ubr013067@bro-ids.icir.org> Message-ID: <20110722233159.GA75115@icir.org> On Fri, Jul 22, 2011 at 16:03 -0700, git at icir.org wrote: > To make it insensitive to scripts loaded by default, now it just > checks that the loaded_scripts log is created and has contents. Not sure about this one. Isn't it actually interesting if the default set changes? And once things are getting stable that shouldn't happen too often. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Fri Jul 22 17:16:20 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 23 Jul 2011 00:16:20 -0000 Subject: [Bro-Dev] #508: ASCII Log Writer precision of time values varies In-Reply-To: <048.2c96fb316543c967f2184acb450ad82b@tracker.bro-ids.org> References: <048.2c96fb316543c967f2184acb450ad82b@tracker.bro-ids.org> Message-ID: <063.07aa2d65aa069f9f3a138aac7b8c577c@tracker.bro-ids.org> #508: ASCII Log Writer precision of time values varies ------------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Feature Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: fixed | Keywords: ------------------------------+------------------------ Comment (by robin): In [b05c34ed29c4bc6b48eb3ce5caca941bc75157a9/bro]: {{{ #!CommitTicketReference repository="bro" revision="b05c34ed29c4bc6b48eb3ce5caca941bc75157a9" LogWriterAscii now prints time values w/ constant 6 digit precision. closes #508 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Jul 22 17:16:20 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 23 Jul 2011 00:16:20 -0000 Subject: [Bro-Dev] #507: topic/jsiwek/pybroccoli-float-repr In-Reply-To: <048.dc0a250607db12c19285690426bd3061@tracker.bro-ids.org> References: <048.dc0a250607db12c19285690426bd3061@tracker.bro-ids.org> Message-ID: <063.f8d4f0a56a0cc7e626d4e6c7ca904438@tracker.bro-ids.org> #507: topic/jsiwek/pybroccoli-float-repr ----------------------------+------------------------ Reporter: jsiwek | Owner: robin Type: Merge Request | Status: closed Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------------+------------------------ Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [4a15b8668f881fdf7845657b80748866db6de749/bro]: {{{ #!CommitTicketReference repository="bro" revision="4a15b8668f881fdf7845657b80748866db6de749" Merge remote-tracking branch 'origin/topic/jsiwek/pybroccoli-float-repr' * origin/topic/jsiwek/pybroccoli-float-repr: Update pybroccoli unit test baseline. Closes #507. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Jul 22 17:33:44 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 23 Jul 2011 00:33:44 -0000 Subject: [Bro-Dev] #472: Cleanup scripts in bro-aux In-Reply-To: <047.3f213fc8d7000dd46ed9e7cf4dcc81a0@tracker.bro-ids.org> References: <047.3f213fc8d7000dd46ed9e7cf4dcc81a0@tracker.bro-ids.org> Message-ID: <062.3d94b09a2ad21775ad845f38d67bcae1@tracker.bro-ids.org> #472: Cleanup scripts in bro-aux ---------------------+--------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: preview ---------------------+--------------------- Comment (by robin): Agree with all you wrote. But I'm thinking we should remove even those which might still work. host-to-addrs and lock_file aren't Bro-specific. ip-grep is about the only one but not sure if anybody is using that. So, anybody wants to keep any of the scripts? (At some point we should develop a new set of scripts that work with the new log files.) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Jul 22 17:40:29 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 23 Jul 2011 00:40:29 -0000 Subject: [Bro-Dev] #511: Misc distribution cleanup Message-ID: <047.7fad0a85cd281981d5bd58a78ff6a111@tracker.bro-ids.org> #511: Misc distribution cleanup ---------------------+-------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: preview | ---------------------+-------------------- There are some things that we need to clean up across all modules: - Many files still have Subversion's $Id$ in them. Delete. - Source files aren't consistent in how they mention their license. I'm thinking we should just a single line statement that refers to the COPYING file. - Update and unifty READMEs -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Fri Jul 22 18:15:27 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Fri, 22 Jul 2011 20:15:27 -0500 Subject: [Bro-Dev] [Bro-Commits] [git/bro] fastpath: Change policy.misc.loaded-scripts tests. (cd5c564) In-Reply-To: <20110722233159.GA75115@icir.org> References: <201107222303.p6MN3ubr013067@bro-ids.icir.org> <20110722233159.GA75115@icir.org> Message-ID: <6179BA86-BCE2-4782-B31D-D7FE47D50897@ncsa.illinois.edu> >> To make it insensitive to scripts loaded by default, now it just >> checks that the loaded_scripts log is created and has contents. > > Not sure about this one. Isn't it actually interesting if the default > set changes? And once things are getting stable that shouldn't happen > too often. It was just bugging the heck out of me seeing tests keep failing that way, but I agree that it's good indicator of a problem if one person's default set of loaded scripts differs from another. I'll put it back so that we have one test that covers that (not a complete revert because old was just testing the event directly and not the provided policy script). - Jon From bro at tracker.bro-ids.org Fri Jul 22 18:36:41 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 23 Jul 2011 01:36:41 -0000 Subject: [Bro-Dev] #472: Cleanup scripts in bro-aux In-Reply-To: <047.3f213fc8d7000dd46ed9e7cf4dcc81a0@tracker.bro-ids.org> References: <047.3f213fc8d7000dd46ed9e7cf4dcc81a0@tracker.bro-ids.org> Message-ID: <062.71ab6633692f3d0e767c7046fdae756f@tracker.bro-ids.org> #472: Cleanup scripts in bro-aux ---------------------+--------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: preview ---------------------+--------------------- Comment (by seth): > So, anybody wants to keep any of the scripts? I've never used any of them. -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Fri Jul 22 20:19:48 2011 From: seth at icir.org (Seth Hall) Date: Fri, 22 Jul 2011 23:19:48 -0400 Subject: [Bro-Dev] mask_addr? Message-ID: <1B74EF7B-92AC-4EB4-A419-9FE8FE5D4CC8@icir.org> I think the existing mask_addr function may be named wrong. Here's the current prototype: function mask_addr(a: addr, top_bits_to_keep: count): addr I want it to do this (and I think it makes more sense based on the name): function mask_addr(a: addr, top_bits_to_keep: count): subnet I'm not sure how the existing function was ever used, but it seems like it must have been a fairly limited use case. Does that seem reasonable to change what that function does and steal the name? I suppose the existing function could be renamed to something else if someone still knows of a use for it. In case it's not obvious from the prototypes above, what I'm aiming to do is take addresses and mask them off to subnets (I need it for aggregation with the metrics framework). For example... print mask_addr(1.2.3.4, 24) => 1.2.3.0/24 .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From vern at icir.org Fri Jul 22 20:23:04 2011 From: vern at icir.org (Vern Paxson) Date: Fri, 22 Jul 2011 20:23:04 -0700 Subject: [Bro-Dev] mask_addr? In-Reply-To: <1B74EF7B-92AC-4EB4-A419-9FE8FE5D4CC8@icir.org> (Fri, 22 Jul 2011 23:19:48 EDT). Message-ID: <20110723032305.02F2236A016@taffy.ICSI.Berkeley.EDU> > I want it to do this (and I think it makes more sense based on the name): > function mask_addr(a: addr, top_bits_to_keep: count): subnet Yeah, it predates the introduction of subnets into Bro. > I'm not sure how the existing function was ever used It was things like: if ( mask_addr(c$id$orig_h, 24) == 1.2.3.0 ) # Whoops, it's coming from 1.2.3/24 ... See {backdoor,ftp,scan}.bro (at least, the 1.5 versions :-) for such uses. > but it seems like it must have been a fairly limited use case. Does > that seem reasonable to change what that function does and steal the name? Yes. Vern From seth at icir.org Fri Jul 22 20:24:58 2011 From: seth at icir.org (Seth Hall) Date: Fri, 22 Jul 2011 23:24:58 -0400 Subject: [Bro-Dev] mask_addr? In-Reply-To: <20110723032305.02F2236A016@taffy.ICSI.Berkeley.EDU> References: <20110723032305.02F2236A016@taffy.ICSI.Berkeley.EDU> Message-ID: On Jul 22, 2011, at 11:23 PM, Vern Paxson wrote: >> I want it to do this (and I think it makes more sense based on the name): >> function mask_addr(a: addr, top_bits_to_keep: count): subnet > > Yeah, it predates the introduction of subnets into Bro. Ah, that would be why I couldn't find anything about it in the CHANGES file. :) I'll file a ticket. Thanks. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Fri Jul 22 20:26:33 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 23 Jul 2011 03:26:33 -0000 Subject: [Bro-Dev] #512: Make the mask_addr function return a subnet Message-ID: <046.90c2b99352e56268367772022621b3e4@tracker.bro-ids.org> #512: Make the mask_addr function return a subnet -----------------------------+-------------------- Reporter: seth | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: preview | -----------------------------+-------------------- My email from the bro-dev list. {{{ I think the existing mask_addr function may be named wrong. Here's the current prototype: function mask_addr(a: addr, top_bits_to_keep: count): addr I want it to do this (and I think it makes more sense based on the name): function mask_addr(a: addr, top_bits_to_keep: count): subnet I'm not sure how the existing function was ever used, but it seems like it must have been a fairly limited use case. Does that seem reasonable to change what that function does and steal the name? I suppose the existing function could be renamed to something else if someone still knows of a use for it. In case it's not obvious from the prototypes above, what I'm aiming to do is take addresses and mask them off to subnets (I need it for aggregation with the metrics framework). For example... print mask_addr(1.2.3.4, 24) => 1.2.3.0/24 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Jul 23 16:39:42 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 23 Jul 2011 23:39:42 -0000 Subject: [Bro-Dev] #472: Cleanup scripts in bro-aux In-Reply-To: <047.3f213fc8d7000dd46ed9e7cf4dcc81a0@tracker.bro-ids.org> References: <047.3f213fc8d7000dd46ed9e7cf4dcc81a0@tracker.bro-ids.org> Message-ID: <062.4e8c2ffe6882e097f593a764f754df1a@tracker.bro-ids.org> #472: Cleanup scripts in bro-aux ---------------------+--------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: preview ---------------------+--------------------- Comment (by vern): I use several of these for my own log analysis/crunching, but I don't have a problem with just maintaining my own copies. If the perception is that Bro users in general don't use these, then removing them strikes me as fine. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Jul 23 16:58:06 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 23 Jul 2011 23:58:06 -0000 Subject: [Bro-Dev] #512: Make the mask_addr function return a subnet In-Reply-To: <046.90c2b99352e56268367772022621b3e4@tracker.bro-ids.org> References: <046.90c2b99352e56268367772022621b3e4@tracker.bro-ids.org> Message-ID: <061.1e6c2414ee08555fc41415fd0082bfc4@tracker.bro-ids.org> #512: Make the mask_addr function return a subnet ------------------------------+--------------------- Reporter: seth | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: preview ------------------------------+--------------------- Comment (by jsiwek): In [b6a371f32d0292a01151f46e11847e3e7a8e440b/bro]: {{{ #!CommitTicketReference repository="bro" revision="b6a371f32d0292a01151f46e11847e3e7a8e440b" mask_addr bif now returns subnet (addresses #512) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Jul 23 19:05:35 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 24 Jul 2011 02:05:35 -0000 Subject: [Bro-Dev] #513: topic/jsiwek/update-restdoc-target Message-ID: <048.bd09641b58156089ec10e0d3ec503dd8@tracker.bro-ids.org> #513: topic/jsiwek/update-restdoc-target ---------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ - adds a documentation coverage test - documentation targets can now be automatically generated w/ a script - fixes for relative @loads - policy script changes in places where I noticed errors being reported -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Jul 24 15:38:43 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 24 Jul 2011 22:38:43 -0000 Subject: [Bro-Dev] #513: topic/jsiwek/update-restdoc-target In-Reply-To: <048.bd09641b58156089ec10e0d3ec503dd8@tracker.bro-ids.org> References: <048.bd09641b58156089ec10e0d3ec503dd8@tracker.bro-ids.org> Message-ID: <063.d323af7bdce6c9250a883ec11eec0902@tracker.bro-ids.org> #513: topic/jsiwek/update-restdoc-target ----------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by robin): * owner: => jsiwek * status: new => assigned Comment: Questions about some of the script changes in here: - you added many @loads. Are they necessary only so that the doc generation works and not otherwise because normally the scripts would be pulled in implicitly? If indeed so, this is not so great: we will probably run into the problem all the time because it will only show up when docs are generated. However, I don't have a good idea right now how to address that. - The change from ProtocolFound to Protocol_Found: is that according to our new naming conventions? - One chunk adds a commit `#This should be optional`: why isn't it? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Jul 25 03:57:05 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 25 Jul 2011 10:57:05 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: jpl Message-ID: <20110725105705.1959D36A3A5@taffy.ICSI.Berkeley.EDU> New user registration for user jpl -- Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Jul 25 07:31:25 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 25 Jul 2011 14:31:25 -0000 Subject: [Bro-Dev] #513: topic/jsiwek/update-restdoc-target In-Reply-To: <048.bd09641b58156089ec10e0d3ec503dd8@tracker.bro-ids.org> References: <048.bd09641b58156089ec10e0d3ec503dd8@tracker.bro-ids.org> Message-ID: <063.c85819b65e468795bdd365f5c8c1662b@tracker.bro-ids.org> #513: topic/jsiwek/update-restdoc-target ----------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by jsiwek): > - you added many @loads. Are they necessary only so that the doc generation works and not otherwise because normally the scripts would be pulled in implicitly? If indeed so, this is not so great: we will probably run into the problem all the time because it will only show up when docs are generated. However, I don't have a good idea right now how to address that. No, the addition of those @loads fixed errors that show up irrespective of enabling documentation mode. > - The change from ProtocolFound to Protocol_Found: is that according to our new naming conventions? That's the format that was used in most places, but the variable naming convention ticket, #484, doesn't seem to have a special rule for Notices (enums are actually supposed to be all caps). Going to follow up in that ticket.. > - One chunk adds a commit `#This should be optional`: why isn't it? I didn't add or delete that comment, I just changed from "##"-style comment to a regular "#" comment because the former has no meaning there and just outputs a warning to that effect. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Jul 25 07:35:23 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 25 Jul 2011 14:35:23 -0000 Subject: [Bro-Dev] #484: Capitalization standardization In-Reply-To: <047.b75bf15c0baa269c752add9d953ccd04@tracker.bro-ids.org> References: <047.b75bf15c0baa269c752add9d953ccd04@tracker.bro-ids.org> Message-ID: <062.652082508370c08327d2c553cb2c606e@tracker.bro-ids.org> #484: Capitalization standardization ----------------------+--------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Website | Version: Resolution: | Keywords: preview ----------------------+--------------------- Changes (by jsiwek): * keywords: => preview Comment: Is the Notice a special case of enum that should use camel casing & underscores? Current examples: Off_Port_Protocol_Found, Malware_Hash_Registry_Match, Site_Exec_Success Also, I think we should try to normalize as much as the naming before the preview as we can, since that's a pretty user visible change to do it later. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Jul 25 07:48:50 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 25 Jul 2011 14:48:50 -0000 Subject: [Bro-Dev] #484: Capitalization standardization In-Reply-To: <047.b75bf15c0baa269c752add9d953ccd04@tracker.bro-ids.org> References: <047.b75bf15c0baa269c752add9d953ccd04@tracker.bro-ids.org> Message-ID: <062.270e194a3ced917d1d4d0f5e97e76eaf@tracker.bro-ids.org> #484: Capitalization standardization ----------------------+--------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Website | Version: Resolution: | Keywords: preview ----------------------+--------------------- Comment (by seth): > Is the Notice a special case of enum that should use camel casing & > underscores? > > Current examples: Off_Port_Protocol_Found, Malware_Hash_Registry_Match, > Site_Exec_Success I wrote that into the draft standard thing I wrote. I've been debating it ever since. Do you have any thoughts on it? Normally, the standard says to do enum values in all upper case with underscores. > Also, I think we should try to normalize as much as the naming before the > preview as we can, since that's a pretty user visible change to do it > later. Agreed. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Jul 25 08:11:34 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 25 Jul 2011 15:11:34 -0000 Subject: [Bro-Dev] #484: Capitalization standardization In-Reply-To: <047.b75bf15c0baa269c752add9d953ccd04@tracker.bro-ids.org> References: <047.b75bf15c0baa269c752add9d953ccd04@tracker.bro-ids.org> Message-ID: <062.eb17ee56639db4bf1f4913eb941ca52c@tracker.bro-ids.org> #484: Capitalization standardization ----------------------+--------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Website | Version: Resolution: | Keywords: preview ----------------------+--------------------- Comment (by jsiwek): > I wrote that into the draft standard thing I wrote. I've been debating it ever since. Do you have any thoughts on it? Seems fine to me, we just need to make the decision and use that format consistently. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Jul 25 08:17:53 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 25 Jul 2011 15:17:53 -0000 Subject: [Bro-Dev] #484: Capitalization standardization In-Reply-To: <047.b75bf15c0baa269c752add9d953ccd04@tracker.bro-ids.org> References: <047.b75bf15c0baa269c752add9d953ccd04@tracker.bro-ids.org> Message-ID: <062.88293fc99f777e19310af44ef66fd148@tracker.bro-ids.org> #484: Capitalization standardization ----------------------+--------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Website | Version: Resolution: | Keywords: preview ----------------------+--------------------- Comment (by seth): > Seems fine to me, we just need to make the decision and use that format > consistently. Yep, I haven't been as consistent with it as I should have. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Jul 25 08:18:46 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 25 Jul 2011 15:18:46 -0000 Subject: [Bro-Dev] #513: topic/jsiwek/update-restdoc-target In-Reply-To: <048.bd09641b58156089ec10e0d3ec503dd8@tracker.bro-ids.org> References: <048.bd09641b58156089ec10e0d3ec503dd8@tracker.bro-ids.org> Message-ID: <063.60bfbab75993c664e429e43ea286fb12@tracker.bro-ids.org> #513: topic/jsiwek/update-restdoc-target ----------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by robin): On Mon, Jul 25, 2011 at 14:31 -0000, you wrote: > No, the addition of those @loads fixed errors that show up irrespective of > enabling documentation mode. Oh, ok, that means those scripts were not working at all? The test-all test should catch such things and we should make sure that it always passes. > I didn't add or delete that comment, I just changed from "##"-style > comment to a regular "#" comment because the former has no meaning there > and just outputs a warning to that effect. I must have misread the diff then, sorry, I thought you added that. Robin -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Jul 25 08:42:17 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 25 Jul 2011 15:42:17 -0000 Subject: [Bro-Dev] #513: topic/jsiwek/update-restdoc-target In-Reply-To: <048.bd09641b58156089ec10e0d3ec503dd8@tracker.bro-ids.org> References: <048.bd09641b58156089ec10e0d3ec503dd8@tracker.bro-ids.org> Message-ID: <063.d9d7e0e45e36aab4b61796b05572e665@tracker.bro-ids.org> #513: topic/jsiwek/update-restdoc-target ----------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by jsiwek): > Oh, ok, that means those scripts were not working at all? At least when loaded "standalone" without any other scripts, the ones I touched all generated varying error messages. > The test-all test should catch such things and we should make sure that it always passes. Yeah, I think so, it's just that's not the test I chose to fix first :) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Jul 25 22:07:55 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 26 Jul 2011 05:07:55 -0000 Subject: [Bro-Dev] #513: topic/jsiwek/update-restdoc-target In-Reply-To: <048.bd09641b58156089ec10e0d3ec503dd8@tracker.bro-ids.org> References: <048.bd09641b58156089ec10e0d3ec503dd8@tracker.bro-ids.org> Message-ID: <063.a47e5c8442c4b2ef3b4ac31bd6946177@tracker.bro-ids.org> #513: topic/jsiwek/update-restdoc-target ----------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------------+------------------------ Changes (by robin): * status: assigned => closed * resolution: => fixed Comment: In [28513b8325060484b636b93ca3e2b7c7759b7a14/bro]: {{{ #!CommitTicketReference repository="bro" revision="28513b8325060484b636b93ca3e2b7c7759b7a14" Merge remote-tracking branch 'origin/topic/jsiwek/update-restdoc-target' * origin/topic/jsiwek/update-restdoc-target: Adding a documentation coverage test. Fixes to relative @load'ing. Closes #513. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Tue Jul 26 15:42:56 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Tue, 26 Jul 2011 17:42:56 -0500 Subject: [Bro-Dev] Quickstart guide feedback In-Reply-To: <20110722162642.GL28458@icir.org> References: <20110722162642.GL28458@icir.org> Message-ID: > Taking a closer look at the quickstart guide was still on my todo > list. It's very nice! but some thoughts about structure: > ... > Taking these together, what I would suggest I think is to actually > have one section just on BroControl, with the corresponding parts > taken out of the current running/configuraion sections; and then a > separate section on just command-line usage. Does that make sense? Yeah, it's more BroControl-centric now and the customization discussion focuses on working through a more specific example/use-case. Let me know if anyone has more ideas about anything that might be added. > - Don't remember whether we talked abot this already, but > navigation links between the sections would be helpful. I remembered it was also mentioned that having it on a single page might be preferred, so I did that for now to see what the impression is. > - For the preview, we should add a note that binary packages won't > be available before the final release. I didn't add a note to the document yet, maybe that's something that could just be pointed out during the preview/beta announcement? - Jon From vallentin at icir.org Wed Jul 27 11:06:10 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Wed, 27 Jul 2011 11:06:10 -0700 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <20110722042053.GB32586@icir.org> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> <20110707215133.GF77684@icir.org> <4E163B71.6050804@ohio.edu> <20110708132228.GA84600@icir.org> <1310914521-sup-1714@samurai.icir.org> <20110722042053.GB32586@icir.org> Message-ID: <1311788385-sup-6149@samurai.icir.org> > However, my main concern is that I don't want to introduce such > functionality through the backdoor by starting to use it here and > there. That leads to code that's really hard to maintain because one > suddenly has a mix of different styles, idioms, level of abstractions, > etc. I fully agree. > For Bro, we had this in the past already with STL containers: they > slowly started to being used more and more, and now we have quite a > mix all over the code. I like the STL containers and I certainly > prefer to use them over the macro-based alternatives (and I do use > them where I can these days), but I still don't think that having both > versions in the same code base is a good thing. Assuming there is not performance/functionality issue, is it too much of an effort to replace the ancient containers? > So, generally, I'd be fine switching to using more modern stuff but I > think that needs to be a conscious decision and we need to adapt the > existing code base accordingly. Eventually, it could be worth going through the new list of language and library features, compare them to the Status Quo, and decide if a change makes sense. For example, there could be smart pointer branch that replaces (Un)Ref() calls with std::shared_ptr. > We also need to make sure we have the platform support everywhere we > need it; don't think we are there yet (or are we?). At least all platforms that support gcc 4.7+ should have no trouble, but I'm sure it will still take a while until we have solid C++11 compilers that are stable on all platforms. > Coming back to the original question, in my view triggering all this > is not really justified just for doing threaded loggging. Longer term, > it may be fine. For just threaded logging, I agree. But this seems to be a rather general questions. I believe that just enhancing one component of Bro with thread support can lead to a problematic architecture in the long run, when other components also jump on the concurrency train. If this is not a coordinated effort, the components may crossfire and impede themselves. This could be as simple as thread thrashing but als manifest more subtle with poor caching and memory performance. Maybe it's not warranted at this point, but I think we should at some point think about a task-based approach to concurrency, rather than thinking about concurrency in terms of threads. (This is completely independent of the multi-core Bro effort but rather about general software engineering practices in many-core land.) > we'd need to argue about it and potentially find arguments for *not* > using it; which will be hard I'm sure. Or somebody might just start > using it and "will change that later" ... Fair enough. But at the end of the day we have to make a decision about what platform-independent network and I/O framework to use. The Boost networking library is of high quality and used widely. If we find a slimmer alternative, that would be great. Matthias From bro at tracker.bro-ids.org Wed Jul 27 12:11:00 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 27 Jul 2011 19:11:00 -0000 Subject: [Bro-Dev] #512: Make the mask_addr function return a subnet In-Reply-To: <046.90c2b99352e56268367772022621b3e4@tracker.bro-ids.org> References: <046.90c2b99352e56268367772022621b3e4@tracker.bro-ids.org> Message-ID: <061.080905b9297ad2140d40ba4443409ad5@tracker.bro-ids.org> #512: Make the mask_addr function return a subnet ----------------------------+--------------------- Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: preview ----------------------------+--------------------- Changes (by jsiwek): * type: Feature Request => Merge Request Comment: This is in `topic/jsiwek/mask_addr_rval` -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jul 27 12:21:10 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 27 Jul 2011 19:21:10 -0000 Subject: [Bro-Dev] #514: NMI scripts under version control Message-ID: <050.07eb55afaed5089f5ed7b9ac51224efd@tracker.bro-ids.org> #514: NMI scripts under version control ----------------------+----------------- Reporter: appleman | Owner: Type: Task | Status: new Priority: Low | Milestone: Component: Bro | Version: Keywords: nmi | ----------------------+----------------- NMI scripts should probably be under revision control * The person fixing something does commits from his local copy after fixing a bug * We should probably use trac tickets so 2 people aren't concurrently fixing things on a shared account on B&T at the same time Using revision control is low priority: it mainly provides redundancy in case NMI content is lost due to some failure on NMI's part -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jul 27 12:33:52 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 27 Jul 2011 19:33:52 -0000 Subject: [Bro-Dev] #515: NMI: Redhat type conversion errors in cxx wrapper made by swig Message-ID: <050.cc968c6768b0f08bb69c74b43963844c@tracker.bro-ids.org> #515: NMI: Redhat type conversion errors in cxx wrapper made by swig -----------------------------------+------------------------ Reporter: appleman | Owner: robin Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: pysubnettree | Version: git/master Keywords: nmi swig pysubnettree | -----------------------------------+------------------------ Redhat is having type conversion errors in the cxx wrapper made by swig {{{ Building CXX object aux/broctl/aux/pysubnettree/CMakeFiles/_SubnetTree.dir/SubnetTreePYTHON_wrap.cxx.o /home/condor/execute/dir_25683/userdir/bro/master/bro/build/aux/broctl/aux/pysubnettree/SubnetTreePYTHON_wrap.cxx: In function 'int SWIG_Python_ConvertFunctionPtr(PyObject*, void**, swig_type_info*)': /home/condor/execute/dir_25683/userdir/bro/master/bro/build/aux/broctl/aux/pysubnettree/SubnetTreePYTHON_wrap.cxx:2051: error: invalid conversion from 'const char*' to 'char*' /home/condor/execute/dir_25683/userdir/bro/master/bro/build/aux/broctl/aux/pysubnettree/SubnetTreePYTHON_wrap.cxx: In function 'int SWIG_AsCharPtrAndSize(PyObject*, char**, size_t*, int*)': /home/condor/execute/dir_25683/userdir/bro/master/bro/build/aux/broctl/aux/pysubnettree/SubnetTreePYTHON_wrap.cxx:2575: error: cannot convert 'int*' to 'Py_ssize_t*' for argument '3' to 'int PyString_AsStringAndSize(PyObject*, char**, Py_ssize_t*)' /home/condor/execute/dir_25683/userdir/bro/master/bro/build/aux/broctl/aux/pysubnettree/SubnetTreePYTHON_wrap.cxx: In function 'void SWIG_Python_FixMethods(PyMethodDef*, swig_const_info*, swig_type_info**, swig_type_info**)': /home/condor/execute/dir_25683/userdir/bro/master/bro/build/aux/broctl/aux/pysubnettree/SubnetTreePYTHON_wrap.cxx:4141: error: invalid conversion from 'const char*' to 'char*' make[2]: *** [aux/broctl/aux/pysubnettree/CMakeFiles/_SubnetTree.dir/SubnetTreePYTHON_wrap.cxx.o] Error 1 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jul 27 12:38:52 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 27 Jul 2011 19:38:52 -0000 Subject: [Bro-Dev] #516: NMI: Output formatting issues in the istate.pybroccoli unit test (Fedora 14) Message-ID: <050.8a31ed622b642173bdfa95bbe6b94396@tracker.bro-ids.org> #516: NMI: Output formatting issues in the istate.pybroccoli unit test (Fedora 14) --------------------------------------+------------------------ Reporter: appleman | Owner: robin Type: Problem | Status: new Priority: Normal | Milestone: Component: broccoli-python | Version: git/master Keywords: nmi btest testing fedora | --------------------------------------+------------------------ There are output formatting issues in the istate.pybroccoli unit test (Fedora 14) that cause the test to fail. Fedora 14 is not currently building successfully on NMI due to apparent NMI issues, so I cannot include the output that shows this problem. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jul 27 12:41:33 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 27 Jul 2011 19:41:33 -0000 Subject: [Bro-Dev] #517: MacOS unit test failures because of missing 'setsid' Message-ID: <050.3adeffdd175bfbc733a2c922a71c552a@tracker.bro-ids.org> #517: MacOS unit test failures because of missing 'setsid' -------------------------------------+------------------------ Reporter: appleman | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: git/master Keywords: nmi btest testing macos | -------------------------------------+------------------------ MacOS unit test failures because of missing 'setsid' command Comment from Adam: I think some repo on NMI just needs to be updated, btest should install a wrapper script that gets used instead of 'setsid' Comment from Don: The "master" git repo on NMI is updated before each build/test run. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jul 27 12:43:07 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 27 Jul 2011 19:43:07 -0000 Subject: [Bro-Dev] #517: NMI: MacOS unit test failures because of missing 'setsid' (was: MacOS unit test failures because of missing 'setsid') In-Reply-To: <050.3adeffdd175bfbc733a2c922a71c552a@tracker.bro-ids.org> References: <050.3adeffdd175bfbc733a2c922a71c552a@tracker.bro-ids.org> Message-ID: <065.59b3339cf84525ad4362a4e454e9c790@tracker.bro-ids.org> #517: NMI: MacOS unit test failures because of missing 'setsid' -----------------------+------------------------------------- Reporter: appleman | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: git/master Resolution: | Keywords: nmi btest testing macos -----------------------+------------------------------------- -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jul 27 12:45:50 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 27 Jul 2011 19:45:50 -0000 Subject: [Bro-Dev] #518: NMI: Use consistent version of CMake Message-ID: <050.2fd3ee8f00f4300b2ccd6054beddb68b@tracker.bro-ids.org> #518: NMI: Use consistent version of CMake -----------------------------+------------------------ Reporter: appleman | Owner: Type: Task | Status: new Priority: Normal | Milestone: Component: Bro | Version: git/master Keywords: nmi build cmake | -----------------------------+------------------------ We should use a consistent version of CMake (the current latest 2.8.4) for building binary packages. This might entail building it from source. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jul 27 12:59:06 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 27 Jul 2011 19:59:06 -0000 Subject: [Bro-Dev] #516: NMI: Output formatting issues in the istate.pybroccoli unit test (Fedora 14) In-Reply-To: <050.8a31ed622b642173bdfa95bbe6b94396@tracker.bro-ids.org> References: <050.8a31ed622b642173bdfa95bbe6b94396@tracker.bro-ids.org> Message-ID: <065.fd74f81d664d5113bb1e222333e2a45d@tracker.bro-ids.org> #516: NMI: Output formatting issues in the istate.pybroccoli unit test (Fedora 14) ------------------------------+-------------------------------------- Reporter: appleman | Owner: robin Type: Problem | Status: new Priority: Normal | Milestone: Component: broccoli-python | Version: git/master Resolution: | Keywords: nmi btest testing fedora ------------------------------+-------------------------------------- Comment (by jsiwek): If this test now passes once the Fedora 14 build is working again, it was probably fixed by #507. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jul 27 13:05:19 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 27 Jul 2011 20:05:19 -0000 Subject: [Bro-Dev] #517: NMI: MacOS unit test failures because of missing 'setsid' In-Reply-To: <050.3adeffdd175bfbc733a2c922a71c552a@tracker.bro-ids.org> References: <050.3adeffdd175bfbc733a2c922a71c552a@tracker.bro-ids.org> Message-ID: <065.07ca3659bb99abcddcae92c7303f1ecb@tracker.bro-ids.org> #517: NMI: MacOS unit test failures because of missing 'setsid' -----------------------+------------------------------------- Reporter: appleman | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: git/master Resolution: | Keywords: nmi btest testing macos -----------------------+------------------------------------- Comment (by jsiwek): Instead of just doing a `git pull` to update the bro repo before each build/test job, it should also be checking that submodules that can be updated do so with `git submodule update --recursive` -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jul 27 13:48:30 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 27 Jul 2011 20:48:30 -0000 Subject: [Bro-Dev] #515: NMI: Redhat type conversion errors in cxx wrapper made by swig In-Reply-To: <050.cc968c6768b0f08bb69c74b43963844c@tracker.bro-ids.org> References: <050.cc968c6768b0f08bb69c74b43963844c@tracker.bro-ids.org> Message-ID: <065.7a2819c4b6a57a174c72b9c477255d6b@tracker.bro-ids.org> #515: NMI: Redhat type conversion errors in cxx wrapper made by swig ---------------------------+----------------------------------- Reporter: appleman | Owner: robin Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: pysubnettree | Version: git/master Resolution: | Keywords: nmi swig pysubnettree ---------------------------+----------------------------------- Comment (by jsiwek): This may be a swig bug: http://permalink.gmane.org/gmane.comp.programming.swig.devel/17840 building with a newer version of swig may be the right answer. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jul 27 14:27:24 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 27 Jul 2011 21:27:24 -0000 Subject: [Bro-Dev] #515: NMI: Redhat type conversion errors in cxx wrapper made by swig In-Reply-To: <050.cc968c6768b0f08bb69c74b43963844c@tracker.bro-ids.org> References: <050.cc968c6768b0f08bb69c74b43963844c@tracker.bro-ids.org> Message-ID: <065.59d1c8d6583ca0d95693169548fd16a9@tracker.bro-ids.org> #515: NMI: Redhat type conversion errors in cxx wrapper made by swig ---------------------------+----------------------------------- Reporter: appleman | Owner: robin Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: pysubnettree | Version: git/master Resolution: | Keywords: nmi swig pysubnettree ---------------------------+----------------------------------- Comment (by jsiwek): I was able to reproduce this with swig 1.3.29 and python 2.7.2 (anything 2.5 supposedly will hit the same problem). Using swig 2.0.4 instead works. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jul 27 14:29:07 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 27 Jul 2011 21:29:07 -0000 Subject: [Bro-Dev] #484: Capitalization standardization In-Reply-To: <047.b75bf15c0baa269c752add9d953ccd04@tracker.bro-ids.org> References: <047.b75bf15c0baa269c752add9d953ccd04@tracker.bro-ids.org> Message-ID: <062.5fc716b6ffcfbd17ca38b83eddf96860@tracker.bro-ids.org> #484: Capitalization standardization ----------------------+---------------------- Reporter: robin | Owner: jsiwek Type: Task | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Website | Version: Resolution: | Keywords: preview ----------------------+---------------------- Changes (by jsiwek): * owner: => jsiwek * status: new => assigned -- Ticket URL: Bro Tracker Bro Issue Tracker From gc355804 at ohio.edu Wed Jul 27 14:37:54 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Wed, 27 Jul 2011 14:37:54 -0700 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <1311788385-sup-6149@samurai.icir.org> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> <20110707215133.GF77684@icir.org> <4E163B71.6050804@ohio.edu> <20110708132228.GA84600@icir.org> <1310914521-sup-1714@samurai.icir.org> <20110722042053.GB32586@icir.org> <1311788385-sup-6149@samurai.icir.org> Message-ID: <4E308532.9090205@ohio.edu> >> Coming back to the original question, in my view triggering all this >> is not really justified just for doing threaded loggging. Longer term, >> it may be fine. >> > > For just threaded logging, I agree. But this seems to be a rather > general questions. I believe that just enhancing one component of Bro > with thread support can lead to a problematic architecture in the long > run, when other components also jump on the concurrency train. If this > is not a coordinated effort, the components may crossfire and impede > themselves. Okay, since this came up and I'm working on it, I guess I'll try to address the architecture issue :) Originally, we were discussing using 0mq, which uses a message-based architecture. This struck me as a very clean way to segment a program into threads, and would logically extend rather well to cover other things (e.g. IPC). As such, I borrowed that model. Bro threads, as I built them (read: my fault if this is bad / stupid :), are simply discrete event processors; they accept messages that descend from a parent EventMessage (or is it MessageEvent? can't remember offhand) and are passed to the individual threads through a thread-safe queue. Because there's a large degree of complexity involved with ensuring any individual event can be processed on any thread, especially given that log / flush / rotate messages have possibly complex ordering dependencies to deal with, and further given that a log writer (or, in bro's case, most of the logwriter-related events) should spend the majority of its time blocking for IO, I don't necessarily agree that logging stuff would be a good candidate for task-oriented execution. Re: task-oriented execution for bro in general: this seems like it is already accomplished to a large degree by e.g. hardware that splits packets by the connection they belong to and routes them to the appropriate processing node in the bro cluster. In this case, the hardware connection splitter thingy acts as the task scheduler, the packet acts as a task, and each individual bro instance acts as the thread of execution. If we wanted to see aggregate performance gains, I guess we could write ubro: micro scripting language, rules are entirely resident in a piece of specialized hardware (CUDA?), processes only certain types of packet streams (thus freeing other general-purpose bro instances to handle other stuff). We could even call it BroCluMP (Bro Cluster Multi-Processing) or something. Naturally, in the case such hardware is not available, it might make sense to build a piece of software that could accomplish the same; I don't know enough about bro's cluster architecture to know whether or not something of this nature already exists, however. > This could be as simple as thread thrashing but als manifest > more subtle with poor caching and memory performance. Yes. That said, this seems relevant: http://www.fullduplex.org/humor/2006/10/how-to-shoot-yourself-in-the-foot-in-any-programming-language/ > Maybe it's not > warranted at this point, but I think we should at some point think about > a task-based approach to concurrency, rather than thinking about > concurrency in terms of threads. (This is completely independent of the > multi-core Bro effort but rather about general software engineering > practices in many-core land.) > > See above :) >> we'd need to argue about it and potentially find arguments for *not* >> using it; which will be hard I'm sure. Or somebody might just start >> using it and "will change that later" ... >> > > Fair enough. But at the end of the day we have to make a decision about > what platform-independent network and I/O framework to use. The Boost > networking library is of high quality and used widely. If we find a > slimmer alternative, that would be great. > Just for the record, boost tends to be less of a library and more of a label; once a library has existed for long enough at a high enough level of quality, it'll often roll itself into boost. As such, one of the libraries bro uses could eventually end up as a piece of boost, at which point the team would have to revisit this particular issue. Just my $0.02. I guarantee neither relevance nor quality. --Gilbert From robin at icir.org Wed Jul 27 16:50:24 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 27 Jul 2011 16:50:24 -0700 Subject: [Bro-Dev] Quickstart guide feedback In-Reply-To: References: <20110722162642.GL28458@icir.org> Message-ID: <20110727235024.GM75193@icir.org> On Tue, Jul 26, 2011 at 17:42 -0500, you wrote: > Yeah, it's more BroControl-centric now and the customization > discussion focuses on working through a more specific > example/use-case. Let me know if anyone has more ideas about anything > that might be added. I like it, thanks! I made a few further small tweaks to the structure, see what you think. > I remembered it was also mentioned that having it on a single page > might be preferred, so I did that for now to see what the impression > is. Yeah, I like that better as well. > I didn't add a note to the document yet, maybe that's something that > could just be pointed out during the preview/beta announcement? Makes sense. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Wed Jul 27 20:48:20 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 28 Jul 2011 03:48:20 -0000 Subject: [Bro-Dev] #519: policy/protocols/http/headers.bro only logs client headers Message-ID: <046.5777f3c379efa3c38e8a08a6979a91db@tracker.bro-ids.org> #519: policy/protocols/http/headers.bro only logs client headers ---------------------+-------------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: | ---------------------+-------------------- In Bro 1.5, policy/http-header.bro logs both client and server headers. The new http/headers.bro only logs client headers, which breaks some forms of analysis. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jul 28 08:02:12 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 28 Jul 2011 15:02:12 -0000 Subject: [Bro-Dev] #520: trace-summary can't parse new conn.log Message-ID: <048.1a6560990d4e6ee8693add3a3cac1151@tracker.bro-ids.org> #520: trace-summary can't parse new conn.log ---------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: trace-summary | Version: git/master Keywords: | ---------------------------+------------------------ e.g.: {{{ ./bin/trace-summary -c < logs/current/conn.log Invalid starting time 1311864275.920013 ... }}} broctl's `summarize-connections` postprocessor relies on trace-summary. -- Ticket URL: Bro Tracker Bro Issue Tracker From vallentin at icir.org Thu Jul 28 08:44:47 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Thu, 28 Jul 2011 08:44:47 -0700 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <4E308532.9090205@ohio.edu> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> <20110707215133.GF77684@icir.org> <4E163B71.6050804@ohio.edu> <20110708132228.GA84600@icir.org> <1310914521-sup-1714@samurai.icir.org> <20110722042053.GB32586@icir.org> <1311788385-sup-6149@samurai.icir.org> <4E308532.9090205@ohio.edu> Message-ID: <1311840515-sup-4852@samurai.icir.org> > Okay, since this came up and I'm working on it, I guess I'll try to > address the architecture issue :) Thanks for chiming in. > Originally, we were discussing using 0mq, which uses a message-based > architecture. This struck me as a very clean way to segment a program > into threads, and would logically extend rather well to cover other > things (e.g. IPC). As such, I borrowed that model. I like the message-passing model as well. How do you use the term "thread?" Do you mean a hardware thread (managed by the OS) or a virtual/logic thread (a user-space abstraction)? I am asking because because a general there should be a (close to) 1:1 ratio between available cores and number of user-level threads, mainly to avoid thrashing and increase cache performance. With I/O-bound applications this is of course less of an issue, but nonetheless a prudent software engineering practice in the manycore era. > Because there's a large degree of complexity involved with ensuring > any individual event can be processed on any thread, especially given > that log / flush / rotate messages have possibly complex ordering > dependencies to deal with, and further given that a log writer (or, in > bro's case, most of the logwriter-related events) should spend the > majority of its time blocking for IO, I don't necessarily agree that > logging stuff would be a good candidate for task-oriented execution. You bring up a good point, namely blocking I/O, which I haven't thought of. Just out of curiosity, could all blocking operations replaced with their asynchronous counterparts? I am just asking because I am using Boost Asio in a completely asynchronous fashion. This lends itself well to a task-based architecture with asynchronous "components," each of which have a task queue that accumulates non-interfering function calls. Imagine that each component has some dynamic number of threads, depending on the available number of cores. Let's assume there exists an asynchronous component for each log backend. (Sorry if I am misusing terminology, I'm not completely up-to-date regarding the logging architecture.) If the log/flush/rotate messages are encapsulated as a single task, then the ordering issues would go away, but you still get a "natural" scale-up at event granularity (assuming events can arrive out-of-order) by assigning more threads to a component. Does that make sense? Maybe you use this sort of architecture already?! My point is essentially that there is a difference between tasks and events that have different notions of concurrency. > Re: task-oriented execution for bro in general: this seems like it is > already accomplished to a large degree by e.g. hardware that splits > packets by the connection they belong to and routes them to the > appropriate processing node in the bro cluster. Yeah, one can think about it this way. The only thing that gives me pause is that the term "task" has a very specific, local meaning in parallel computation lingo. > If we wanted to see aggregate performance gains, I guess we could > write ubro: micro scripting language, rules are entirely resident in a > piece of specialized hardware (CUDA?), processes only certain types of > packet streams (thus freeing other general-purpose bro instances to > handle other stuff). Nice thought, that's something for HILTI where available hardware is transparently used by the execution environment. For example, if a hardware regex matcher is available, the execution environment offloads the relevant instructions to the special card but would otherwise use its own implementation. > We could even call it BroCluMP (Bro Cluster Multi-Processing) or > something. While we're creating new names, what about Bruda for the GPU-based version of Bro that offloads regex matching to CUDA? ;-) Matthias From bro at tracker.bro-ids.org Thu Jul 28 08:51:36 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 28 Jul 2011 15:51:36 -0000 Subject: [Bro-Dev] #520: trace-summary can't parse new conn.log In-Reply-To: <048.1a6560990d4e6ee8693add3a3cac1151@tracker.bro-ids.org> References: <048.1a6560990d4e6ee8693add3a3cac1151@tracker.bro-ids.org> Message-ID: <063.a3a7c2eba2bfc282fed30a8f48a0f476@tracker.bro-ids.org> #520: trace-summary can't parse new conn.log ----------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: trace-summary | Version: git/master Resolution: fixed | Keywords: ----------------------------+------------------------ Changes (by jsiwek): * status: new => closed * resolution: => fixed Comment: In [975204aa760b5940aeeb185e0f986680ba7b3333/trace-summary]: {{{ #!CommitTicketReference repository="trace-summary" revision="975204aa760b5940aeeb185e0f986680ba7b3333" Teach trace-summary to parse Bro's new conn.log format (closes #520) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From slagell at ncsa.illinois.edu Thu Jul 28 08:52:03 2011 From: slagell at ncsa.illinois.edu (Adam J. Slagell) Date: Thu, 28 Jul 2011 09:52:03 -0600 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <1311840515-sup-4852@samurai.icir.org> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> <20110707215133.GF77684@icir.org> <4E163B71.6050804@ohio.edu> <20110708132228.GA84600@icir.org> <1310914521-sup-1714@samurai.icir.org> <20110722042053.GB32586@icir.org> <1311788385-sup-6149@samurai.icir.org> <4E308532.9090205@ohio.edu> <1311840515-sup-4852@samurai.icir.org> Message-ID: On Jul 28, 2011, at 9:44 AM, Matthias Vallentin wrote: > I am asking because > because a general there should be a (close to) 1:1 ratio between > available cores and number of user-level threads, mainly to avoid > thrashing and increase cache performance. We don't know how many cores the user will have and how many are virtual (hyperthreading). Due to the direction hardware is heading, I think if we estimate and hard code it in the design, it will soon be an underestimate. Therefore, I would probably err on the side of more threads than is typical in a server today when designing a multi-threaded architecture. ------ Adam J. Slagell, CISSP Sr. Security Engineer National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info 217.244.8965 From vallentin at icir.org Thu Jul 28 09:01:15 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Thu, 28 Jul 2011 09:01:15 -0700 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> <20110707215133.GF77684@icir.org> <4E163B71.6050804@ohio.edu> <20110708132228.GA84600@icir.org> <1310914521-sup-1714@samurai.icir.org> <20110722042053.GB32586@icir.org> <1311788385-sup-6149@samurai.icir.org> <4E308532.9090205@ohio.edu> <1311840515-sup-4852@samurai.icir.org> Message-ID: <1311868522-sup-6034@samurai.icir.org> > We don't know how many cores the user will have and how many are > virtual (hyperthreading). Due to the direction hardware is heading, I > think if we estimate and hard code it in the design, it will soon be > an underestimate. Therefore, I would probably err on the side of more > threads than is typical in a server today when designing a > multi-threaded architecture. One way to get around this is to just have a command line switch that specifies the number of threads to use in total. This would override the default concurrency level as detected by the application (i.e., the number of cores exposed by the OS). The applications than somehow internally distributes the number of available threads over the asynchronous components. In fact, one often uses a slightly tweaked ratio with a little more threads than what the hardware exposes, as it achieves better saturation. (I guess that's what you mean by "err on the side of more threads.") Matthias From gregor at icir.org Thu Jul 28 09:01:50 2011 From: gregor at icir.org (Gregor Maier) Date: Thu, 28 Jul 2011 09:01:50 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/trace-summary] fastpath: Teach trace-summary to parse Bro's new conn.log format (closes #520) (975204a) In-Reply-To: <201107281551.p6SFpauH011943@bro-ids.icir.org> References: <201107281551.p6SFpauH011943@bro-ids.icir.org> Message-ID: <4E3187EE.5060502@icir.org> Hi, I don't know whether ConnSizeAnalyzer's output (number of IP bytes and packets on the wire) can also be added to the new conn.log (Seth?). If so, it might make sense to give trace-summary a flag to use those counters. cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From slagell at ncsa.illinois.edu Thu Jul 28 09:03:18 2011 From: slagell at ncsa.illinois.edu (Adam J. Slagell) Date: Thu, 28 Jul 2011 10:03:18 -0600 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <1311868522-sup-6034@samurai.icir.org> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> <20110707215133.GF77684@icir.org> <4E163B71.6050804@ohio.edu> <20110708132228.GA84600@icir.org> <1310914521-sup-1714@samurai.icir.org> <20110722042053.GB32586@icir.org> <1311788385-sup-6149@samurai.icir.org> <4E308532.9090205@ohio.edu> <1311840515-sup-4852@samurai.icir.org> <1311868522-sup-6034@samurai.icir.org> Message-ID: On Jul 28, 2011, at 10:01 AM, Matthias Vallentin wrote: >> We don't know how many cores the user will have and how many are >> virtual (hyperthreading). Due to the direction hardware is heading, I >> think if we estimate and hard code it in the design, it will soon be >> an underestimate. Therefore, I would probably err on the side of more >> threads than is typical in a server today when designing a >> multi-threaded architecture. > > One way to get around this is to just have a command line switch that > specifies the number of threads to use in total. This would override the > default concurrency level as detected by the application (i.e., the > number of cores exposed by the OS). The applications than somehow > internally distributes the number of available threads over the > asynchronous components. > > In fact, one often uses a slightly tweaked ratio with a little more > threads than what the hardware exposes, as it achieves better > saturation. (I guess that's what you mean by "err on the side of more > threads.") It wasn't clear to me whether or not the number of threads was really defined by the architecture. If there is always more work that you can give to more threads, then this works. From seth at icir.org Thu Jul 28 09:58:38 2011 From: seth at icir.org (Seth Hall) Date: Thu, 28 Jul 2011 12:58:38 -0400 Subject: [Bro-Dev] [Bro-Commits] [git/trace-summary] fastpath: Teach trace-summary to parse Bro's new conn.log format (closes #520) (975204a) In-Reply-To: <4E3187EE.5060502@icir.org> References: <201107281551.p6SFpauH011943@bro-ids.icir.org> <4E3187EE.5060502@icir.org> Message-ID: On Jul 28, 2011, at 12:01 PM, Gregor Maier wrote: > I don't know whether ConnSizeAnalyzer's output (number of IP bytes and > packets on the wire) can also be added to the new conn.log (Seth?). If > so, it might make sense to give trace-summary a flag to use those counters. I still haven't done the ConnSizeAnalyzer integration where I want it yet. Does it seem unreasonable to anyone to have that enabled mostly by default? I think it's useful enough that it would be enabled nearly all the time anyway. I want/need to integrate it into the SSH analysis and Conn analysis at the very least. If we have the ConnSizeAnalyzer enabled "by default" then I'll add the fields to the core conn scripts, otherwise it will be an extension script (I'd rather have it in the core script support). .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From gc355804 at ohio.edu Thu Jul 28 10:50:42 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Thu, 28 Jul 2011 13:50:42 -0400 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <1311840515-sup-4852@samurai.icir.org> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> <20110707215133.GF77684@icir.org> <4E163B71.6050804@ohio.edu> <20110708132228.GA84600@icir.org> <1310914521-sup-1714@samurai.icir.org> <20110722042053.GB32586@icir.org> <1311788385-sup-6149@samurai.icir.org> <4E308532.9090205@ohio.edu> <1311840515-sup-4852@samurai.icir.org> Message-ID: <4E31A172.6010600@ohio.edu> Hi: Inline: >> Originally, we were discussing using 0mq, which uses a message-based >> architecture. This struck me as a very clean way to segment a program >> into threads, and would logically extend rather well to cover other >> things (e.g. IPC). As such, I borrowed that model. > I like the message-passing model as well. How do you use the term > "thread?" Do you mean a hardware thread (managed by the OS) or a > virtual/logic thread (a user-space abstraction)? I am asking because > because a general there should be a (close to) 1:1 ratio between > available cores and number of user-level threads, mainly to avoid > thrashing and increase cache performance. With I/O-bound applications > this is of course less of an issue, but nonetheless a prudent software > engineering practice in the manycore era. > Whichever model pthread happens to use :) I think the implementation might be slightly platform-dependent. That said, keep in mind that some libraries (e.g. DataSeries) actually spawn additional worker threads. This makes it very difficult to place a hard limit on the number of threads that exist within a bro process. If this does turn out to be an issue, it may be time to look at fork()'ing the loggers and / or using some kind of remote logging scheme. >> Because there's a large degree of complexity involved with ensuring >> any individual event can be processed on any thread, especially given >> that log / flush / rotate messages have possibly complex ordering >> dependencies to deal with, and further given that a log writer (or, in >> bro's case, most of the logwriter-related events) should spend the >> majority of its time blocking for IO, I don't necessarily agree that >> logging stuff would be a good candidate for task-oriented execution. > You bring up a good point, namely blocking I/O, which I haven't thought > of. Just out of curiosity, could all blocking operations replaced with > their asynchronous counterparts? For the ASCII and the DataSeries loggers, yes; the latter seems to already do this (one of the worker threads is an output thread). > I am just asking because I am using > Boost Asio in a completely asynchronous fashion. This lends itself well > to a task-based architecture with asynchronous "components," each of > which have a task queue that accumulates non-interfering function calls. > Imagine that each component has some dynamic number of threads, > depending on the available number of cores. Okay. > Let's assume there exists an asynchronous component for each log > backend. (Sorry if I am misusing terminology, I'm not completely > up-to-date regarding the logging architecture.) I think I follow you. > If the log/flush/rotate > messages are encapsulated as a single task, then the ordering issues > would go away, but you still get a "natural" scale-up at event > granularity (assuming events can arrive out-of-order) by assigning more > threads to a component. I don't understand this bit. Say we have 10 things to log: {A, B, C, ... J}, and we queue 8 messages before we flush + rotate (for the sake of argument) Log+Flush+Rotate task = {A, B, C, D} ---> Thread 1 Log+Flush task = {E, F, G, H} ---> Thread 2 Log+Flush+Finish task = {I, J} ---> Thread 3 In this case, it seems possible that thread 3 could complete before threads 1 and 2, unless we forced tasks to lock the log file upon execution. . . but if we did that, I think the locking order would become less predictable as the system's load increased, leading to oddly placed chunks of log data and / or dropped messages if the file were to be closed before a previous rotate message made it to the logger. > Does that make sense? Maybe you use this sort of > architecture already?! Kind of. Let me throw together a diagram and send it out; it's something that should probably end up in the documentation anyway :) > My point is essentially that there is a > difference between tasks and events that have different notions of > concurrency. > I need to do some more reading before I'd be prepared to offer a coherent argument here. >> Re: task-oriented execution for bro in general: this seems like it is >> already accomplished to a large degree by e.g. hardware that splits >> packets by the connection they belong to and routes them to the >> appropriate processing node in the bro cluster. > Yeah, one can think about it this way. The only thing that gives me > pause is that the term "task" has a very specific, local meaning in > parallel computation lingo. > I don't claim to be a parallel computation expert, so this does not surprise me :) I was more attempting to illustrate the parallel between bro's existing cluster architecture and a more traditional task-oriented model (as I understand it) than I was trying to argue that packets should necessarily be classifiable as "tasks" in the strictest sense of the word. >> If we wanted to see aggregate performance gains, I guess we could >> write ubro: micro scripting language, rules are entirely resident in a >> piece of specialized hardware (CUDA?), processes only certain types of >> packet streams (thus freeing other general-purpose bro instances to >> handle other stuff). > Nice thought, that's something for HILTI where available hardware is > transparently used by the execution environment. For example, if a > hardware regex matcher is available, the execution environment offloads > the relevant instructions to the special card but would otherwise use > its own implementation. > Ah, right then. Thanks, Gilbert From vern at icir.org Thu Jul 28 11:47:15 2011 From: vern at icir.org (Vern Paxson) Date: Thu, 28 Jul 2011 11:47:15 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/trace-summary] fastpath: Teach trace-summary to parse Bro's new conn.log format (closes #520) (975204a) In-Reply-To: (Thu, 28 Jul 2011 12:58:38 EDT). Message-ID: <20110728184715.369D236A394@taffy.ICSI.Berkeley.EDU> > I still haven't done the ConnSizeAnalyzer integration where I want it yet. Does it seem unreasonable to anyone to have that enabled mostly by default? I think it's useful enough that it would be enabled nearly all the time anyway. Yeah, it strikes me as bread-&-butter info that it's sort of embarrassing the logs historically didn't track :-P. Vern From vallentin at icir.org Thu Jul 28 12:05:28 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Thu, 28 Jul 2011 12:05:28 -0700 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <4E31A172.6010600@ohio.edu> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> <20110707215133.GF77684@icir.org> <4E163B71.6050804@ohio.edu> <20110708132228.GA84600@icir.org> <1310914521-sup-1714@samurai.icir.org> <20110722042053.GB32586@icir.org> <1311788385-sup-6149@samurai.icir.org> <4E308532.9090205@ohio.edu> <1311840515-sup-4852@samurai.icir.org> <4E31A172.6010600@ohio.edu> Message-ID: <1311878972-sup-3675@samurai.icir.org> > That said, keep in mind that some libraries (e.g. DataSeries) actually > spawn additional worker threads. Is it possible to configure the number of threads DataSeries uses? > I don't understand this bit. Say we have 10 things to log: {A, B, C, > ... J}, and we queue 8 messages before we flush + rotate (for the sake > of argument) > > Log+Flush+Rotate task = {A, B, C, D} ---> Thread 1 > Log+Flush task = {E, F, G, H} ---> Thread 2 > Log+Flush+Finish task = {I, J} ---> Thread 3 > > In this case, it seems possible that thread 3 could complete before > threads 1 and 2, unless we forced tasks to lock the log file upon > execution. . . but if we did that, I think the locking order would > become less predictable as the system's load increased, leading to oddly > placed chunks of log data and / or dropped messages if the file were to > be closed before a previous rotate message made it to the logger. This is a good example of hot to not use task-based parallelism. As there exist inter-task dependencies, they have to be processed sequentially. For example, say these 10 things to log are generated for 18 http_reply events. What I was saying is that you have one task for all dependent actions/messages/steps: Event 1: Log+Flush+Rotate+Log+Flush+Finish task = {A, ..., J} ... Event 18: Log+Flush+Rotate+Log+Flush+Finish task = {A, ..., J} Say the Log Manager component has 3 threads associated, then each thread would process 6 events, assuming a simple uniform scheduling strategy. > Kind of. Let me throw together a diagram and send it out; it's > something that should probably end up in the documentation anyway :) Sounds good. This stuff is much better to comprehend with a visual aid! > I was more attempting to illustrate the parallel between bro's existing > cluster architecture and a more traditional task-oriented model (as I > understand it) than I was trying to argue that packets should > necessarily be classifiable as "tasks" in the strictest sense of the word. Yup, I understood it that way and think it's a nice analogy. Matthias From bro at tracker.bro-ids.org Thu Jul 28 12:56:53 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 28 Jul 2011 19:56:53 -0000 Subject: [Bro-Dev] #484: Capitalization standardization In-Reply-To: <047.b75bf15c0baa269c752add9d953ccd04@tracker.bro-ids.org> References: <047.b75bf15c0baa269c752add9d953ccd04@tracker.bro-ids.org> Message-ID: <062.28c7e991c469c0649e8183f8fe020775@tracker.bro-ids.org> #484: Capitalization standardization ----------------------+--------------------- Reporter: robin | Owner: jsiwek Type: Task | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Website | Version: Resolution: fixed | Keywords: preview ----------------------+--------------------- Changes (by jsiwek): * status: assigned => closed * resolution: => fixed Comment: In [771728720d4f4e97a8906ef59ac45a2167919132/bro]: {{{ #!CommitTicketReference repository="bro" revision="771728720d4f4e97a8906ef59ac45a2167919132" Normalize Notice::Type identifiers per convention. (closes #484) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Thu Jul 28 13:49:03 2011 From: gregor at icir.org (Gregor Maier) Date: Thu, 28 Jul 2011 13:49:03 -0700 Subject: [Bro-Dev] Tunnels Message-ID: <4E31CB3F.3090400@icir.org> Hi, in NetSessions::DispatchPacket() Bro has some code to decapsulate some tunnels. The first one uses encap_hdr_size (and udp_tunnel_port). It appears that it expects an IP header and if it finds one(*) and if encap_hdr_size>0 then it will add skip encap_hdr_size and assume that the "interesting" part of the packet starts there. I was wondering why the code checks whether there is an IP header before skipping the fixed encap_hdr_bytes? Note that there is also the option to only skip if the packet is UDP and has a particular destination port (udp_tunnel_port). For this we do need to look for the IP header.... I think there might be a misplaced "else" here (i.e., the else belongs to a different if) (*) it assumes there is one and just checks whether the caplen is long enough to contain the full IP header according to IP header length. The other way it does tunnel decapsulation is done when parse_udp_tunnels is set. If the packet contains an IP header + UDP header the code checks whether the UDP payload looks like an IP packet and if so, then it will skip over these initial IP+UDP header. (Thus this code decapsulates IP-over-UDP). If udp_tunnel_port is specified then this code will limit its decapsulation to UDP packet with that destination port. Some problems/concerns/questions: * What was the intention of this code? * All this happens before IP fragmentation. Shouldn't this be after defragmentation (and then re-inject the decapsulated packets before the de-fragmentation in the case that the tunneled IP is fragmented as well) * The whole code is hard-code for IPv4 and doesn't check whether the initial IP header is actually a valid IP header (it assumes it is and just used IP header length). For IPv6 or ARP packets this code would obviously be incorrect (and possibly the code would incorrectly decapsulate something it shouldn't) * While the code checks that there is enough packet data for the IP header, it does not check whether there is enough data for the UDP header before accessing uh_len and uh_dport. In the 2nd case it also doesn't check whether there's enough captured data to contain the encapsulated packet (Bro does look at this data for its heuristic that checks whether there is an IP header here) * What should we do with this? Fix? Remove? Modify? Note, the reason I was looking at this code is that I want to write something that can decapsulate tunneled IPv6 packets (6in4, 6to4, Teredo). cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Thu Jul 28 13:53:04 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 28 Jul 2011 20:53:04 -0000 Subject: [Bro-Dev] #521: Bro's secondary path does not handle IPv6) (only v4 Message-ID: <048.aaaca7825f512275e9b84e037a2e0a6b@tracker.bro-ids.org> #521: Bro's secondary path does not handle IPv6) (only v4 ---------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: git/master Keywords: IPv6 | ---------------------+------------------------ Bro's secondary path only handles IPv4 packets See NetSessions::NextPacketSecondary -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jul 28 13:58:46 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 28 Jul 2011 20:58:46 -0000 Subject: [Bro-Dev] #522: Event to report non TCP/UDP/ICMP packets Message-ID: <048.9578ab814d281f028dba6ef156f882b0@tracker.bro-ids.org> #522: Event to report non TCP/UDP/ICMP packets ---------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: git/master Keywords: IPv6 | ---------------------+------------------------ Hi, it might be useful to have an event that fires if an IP packet does not contain TCP, UDP or ICMP, i.e., it either has a different IP protocol number or for IPv6 it might be some sort of extension header. Currently Bro silently ignores these packets. See NetSessions::DoNextPacket() -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jul 28 14:06:20 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 28 Jul 2011 21:06:20 -0000 Subject: [Bro-Dev] #523: event new_packet does not support IPv6 Message-ID: <048.6a57705fb7303481ddab2ff96d7bc20e@tracker.bro-ids.org> #523: event new_packet does not support IPv6 ---------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: git/master Keywords: IPv6 | ---------------------+------------------------ The new_packet event is only raised for IPv4 packets. More generally every event that uses either the pkt_hdr or the ip_hdr records might have the same problem. -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Thu Jul 28 14:36:03 2011 From: gregor at icir.org (Gregor Maier) Date: Thu, 28 Jul 2011 14:36:03 -0700 Subject: [Bro-Dev] PacketSorter Message-ID: <4E31D643.5070208@icir.org> Hi, I was wondering about Bro's packet sorter. From a quick glance it appears that it's only enabled if packet_sort_window is set to a non zero value. When enabled it will sort packets a) based on timestamps and b) for TCP packets based on SEQ/ACK numbers (I presume to ensure that ACKs are delivered after the data packet) Note, this is independent from Bro's ability to process multiple trace files (or multiple interfaces) in order. So I was wondering about the use cases for PacketSorter, especially (a) If the packet sorter is enabled Bro's behavior will slightly change: It won't pass ARP packets to the ARP analyzer, and it won't create a weird if it's not an IP packet. I was just wondering whether anybody has recently used the packet sorter. If not I'm wondering whether we should test this code path to see whether it works correctly esp wrt IPv6. Or, actually, whether the packet sorter is worth keeping or whether we should remove the code. And another question would be if the TCP sorting would better be handled by the TCP analyzer? Opinions? cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Thu Jul 28 14:57:54 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 28 Jul 2011 21:57:54 -0000 Subject: [Bro-Dev] #524: Bro fuzz testing Message-ID: <048.a8b2fd1e8661b692c679d3046e408dc7@tracker.bro-ids.org> #524: Bro fuzz testing --------------------+------------------------ Reporter: gregor | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | --------------------+------------------------ As part of Bro's regular testing we should probably also to some fuzzing. Good starting points might be: * http://wiki.wireshark.org/FuzzTesting * http://code.google.com/p/capouik/ (this one maybe not so much) The wireshark tools can probably be used with very little modification. However, one worthwhile addition might be to make the tools preserve the first n bytes of a packet and then start the fuzzing afterwards (e.g., to only fuzz after the IP header or such). (Not sure whether this should be milestone 1.6 or 1.7) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jul 28 15:27:39 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 28 Jul 2011 22:27:39 -0000 Subject: [Bro-Dev] #509: SMTP rework In-Reply-To: <046.35777c113d149585390ec44632cce689@tracker.bro-ids.org> References: <046.35777c113d149585390ec44632cce689@tracker.bro-ids.org> Message-ID: <061.f2664cc170ce8c1c6dfea2aef84153d4@tracker.bro-ids.org> #509: SMTP rework ---------------------+------------------------ Reporter: seth | Owner: jsiwek Type: Task | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: preview ---------------------+------------------------ Changes (by jsiwek): * owner: => jsiwek * status: new => assigned -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Thu Jul 28 17:28:18 2011 From: seth at icir.org (Seth Hall) Date: Thu, 28 Jul 2011 20:28:18 -0400 Subject: [Bro-Dev] Tunnels In-Reply-To: <4E31CB3F.3090400@icir.org> References: <4E31CB3F.3090400@icir.org> Message-ID: <9E5DC6EA-AB5A-4B10-A2B7-FE999D891EC8@icir.org> On Jul 28, 2011, at 4:49 PM, Gregor Maier wrote: > * What was the intention of this code? There is a lot of guessing here as the code was written before I started working with Bro. The only way I have seen encap_hdr_size used is to remove VLAN headers. That is done properly now so that use of the encap_hdr_size variable is gone. I have no clue what UDP tunnels were being decapsulated though. In my opinion it was probably something that should have been done with an analyzer anyway and is probably ok to remove. > * All this happens before IP fragmentation. Like above, I think it's because it's to remove lower level encapsulation. > * What should we do with this? Fix? Remove? Modify? I say remove. I was tempted to do it when I fixing the VLAN and MPLS handling but I had the same questions as you. > Note, the reason I was looking at this code is that I want to write > something that can decapsulate tunneled IPv6 packets (6in4, 6to4, Teredo). I have code to do this written for Click! and it's fortunately very easy (I have AYIYA decapsulation code too). The problem is that you have two IP layer headers and only the one with your address space makes any sense. It's really confusing to see two non-local networks show up in your conn.log because it was a local host using tunneled address space. I think we need the ability in the connection record to specify a parent connection or something. I suppose even just logging the presence of a tunnel may be enough. There are a lot of questions that will need answered once we start decapsulating tunnels. I say go for it! :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Thu Jul 28 18:13:30 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 29 Jul 2011 01:13:30 -0000 Subject: [Bro-Dev] #512: Make the mask_addr function return a subnet In-Reply-To: <046.90c2b99352e56268367772022621b3e4@tracker.bro-ids.org> References: <046.90c2b99352e56268367772022621b3e4@tracker.bro-ids.org> Message-ID: <061.1d3f9d2bdbf143635093c58b56dff915@tracker.bro-ids.org> #512: Make the mask_addr function return a subnet ----------------------------+--------------------- Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: preview ----------------------------+--------------------- Comment (by robin): In [0e5bc16a60aeb9766fe697f11a44da1d8c2672a0/bro]: {{{ #!CommitTicketReference repository="bro" revision="0e5bc16a60aeb9766fe697f11a44da1d8c2672a0" Merge remote-tracking branch 'origin/topic/jsiwek/mask_addr_rval' * origin/topic/jsiwek/mask_addr_rval: mask_addr bif now returns subnet (addresses #512) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jul 28 18:13:30 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 29 Jul 2011 01:13:30 -0000 Subject: [Bro-Dev] #484: Capitalization standardization In-Reply-To: <047.b75bf15c0baa269c752add9d953ccd04@tracker.bro-ids.org> References: <047.b75bf15c0baa269c752add9d953ccd04@tracker.bro-ids.org> Message-ID: <062.15b7c133fd1a1f56db2d948316d129ab@tracker.bro-ids.org> #484: Capitalization standardization ----------------------+--------------------- Reporter: robin | Owner: jsiwek Type: Task | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Website | Version: Resolution: fixed | Keywords: preview ----------------------+--------------------- Comment (by robin): In [4baf3442786bafec1eaefdb41abf68bf7fc9e5e8/bro]: {{{ #!CommitTicketReference repository="bro" revision="4baf3442786bafec1eaefdb41abf68bf7fc9e5e8" Merge remote-tracking branch 'origin/fastpath' * origin/fastpath: Normalize Notice::Type identifiers per convention. (closes #484) Another fix to the default-loaded-scripts test. Add new piped_exec BiF. Revert "Fixes for email_notice_to() function." Fixes for email_notice_to() function. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Thu Jul 28 18:23:22 2011 From: gregor at icir.org (Gregor Maier) Date: Thu, 28 Jul 2011 18:23:22 -0700 Subject: [Bro-Dev] Tunnels In-Reply-To: <9E5DC6EA-AB5A-4B10-A2B7-FE999D891EC8@icir.org> References: <4E31CB3F.3090400@icir.org> <9E5DC6EA-AB5A-4B10-A2B7-FE999D891EC8@icir.org> Message-ID: <4E320B8A.8060305@icir.org> > [was used for VLAN] Hmm. Interesting. In this case I'm surprised it worked because the way I read the code, it always assumes that there is an IP header. It cast the pkt pointer into a struct ip* and checks whether the caplen is large enough to the IP header length from this struct ip* (which should work often enough because the max header length is 64 bytes) but some short packets might have been missed by that code. >> Note, the reason I was looking at this code is that I want to write >> something that can decapsulate tunneled IPv6 packets (6in4, 6to4, Teredo). > > > I have code to do this written for Click! and it's fortunately very easy (I have AYIYA decapsulation code too). Cool. I've actually never used Click, but yeah, the actual decapsulation is really quite easy. Deciding what to do with the parent connection (and how to implement that) is the hard part.... > The problem is that you have two IP layer headers and only the one with your address space makes any sense. It's really confusing to see two non-local networks show up in your conn.log because it was a local host using tunneled address space. The right way to do that is to pull IP analysis into the analyzer tree and then do tunnel de-capsulation using the analyzer tree. Then you could even decapsulate IP-over-HTTP-over-whatever. But that's going to be a lot of work. My plan was to do a simpler work-around though: > I think we need the ability in the connection record to specify a parent connection or something. I suppose even just logging the presence of a tunnel may be enough. There are a lot of questions that will need answered once we start decapsulating tunnels. > Yeah. I haven't looked into that too much yet. But I was thinking of either (or possibly both configureable) * removing the parent headers completely but generate an event to associate the child connection with the parent connection (1) * create a copy for the packet data of the child . Thus Bro would see both the parent connection and the decapsulated connection. One would still want an event to link the two and you'll have two packets with the same timestamp and the total byte counts will be inflated..... (1) not that it's not necessarily a connection. 6to4 and 6in4 are directly on top of IP (using proto 41). So Bro wouldn't even see these packets because it only handles TCP/UDP/ICMP (you would get a weird though) cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From vern at icir.org Thu Jul 28 18:41:55 2011 From: vern at icir.org (Vern Paxson) Date: Thu, 28 Jul 2011 18:41:55 -0700 Subject: [Bro-Dev] Tunnels In-Reply-To: <4E320B8A.8060305@icir.org> (Thu, 28 Jul 2011 18:23:22 PDT). Message-ID: <20110729014155.998D836A398@taffy.ICSI.Berkeley.EDU> > > [was used for VLAN] > > Hmm. Interesting. In this case I'm surprised it worked because the way I > read the code, it always assumes that there is an IP header. Quick comment: this was an earlier hack to deal with IP-in-UDP tunneling used for our honeynet setup. Vern From gc355804 at ohio.edu Thu Jul 28 19:04:28 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Thu, 28 Jul 2011 19:04:28 -0700 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <1311878972-sup-3675@samurai.icir.org> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> <20110707215133.GF77684@icir.org> <4E163B71.6050804@ohio.edu> <20110708132228.GA84600@icir.org> <1310914521-sup-1714@samurai.icir.org> <20110722042053.GB32586@icir.org> <1311788385-sup-6149@samurai.icir.org> <4E308532.9090205@ohio.edu> <1311840515-sup-4852@samurai.icir.org> <4E31A172.6010600@ohio.edu> <1311878972-sup-3675@samurai.icir.org> Message-ID: <4E32152C.1000407@ohio.edu> Matthias Vallentin wrote: >> That said, keep in mind that some libraries (e.g. DataSeries) actually >> spawn additional worker threads. >> > > Is it possible to configure the number of threads DataSeries uses? > Yes, to some extent; DataSeries will always spawn at least two threads (one output thread and one compression / worker thread). The number of worker threads DataSeries spawns can be controlled by tweaking a variable in dataseries.bro, but this brings up a more general problem: what if we use a library that doesn't offer us the ability to do this? Then again, if the library doesn't let us do this, maybe we just shouldn't be using that particular library...? >> I don't understand this bit. Say we have 10 things to log: {A, B, C, >> ... J}, and we queue 8 messages before we flush + rotate (for the sake >> of argument) >> >> Log+Flush+Rotate task = {A, B, C, D} ---> Thread 1 >> Log+Flush task = {E, F, G, H} ---> Thread 2 >> Log+Flush+Finish task = {I, J} ---> Thread 3 >> >> In this case, it seems possible that thread 3 could complete before >> threads 1 and 2, unless we forced tasks to lock the log file upon >> execution. . . but if we did that, I think the locking order would >> become less predictable as the system's load increased, leading to oddly >> placed chunks of log data and / or dropped messages if the file were to >> be closed before a previous rotate message made it to the logger. >> > > This is a good example of hot to not use task-based parallelism. As > there exist inter-task dependencies, they have to be processed > sequentially. For example, say these 10 things to log are generated for > 18 http_reply events. What I was saying is that you have one task for > all dependent actions/messages/steps: > > Event 1: Log+Flush+Rotate+Log+Flush+Finish task = {A, ..., J} > ... > Event 18: Log+Flush+Rotate+Log+Flush+Finish task = {A, ..., J} > > Say the Log Manager component has 3 threads associated, then each thread > would process 6 events, assuming a simple uniform scheduling strategy. > > Sure. In this case, though, A-J essentially compose the entire log file, which is what's giving me some trouble; even if the LogManager has 3 threads associated, it still has to pass entire LogWriters around between those threads and make sure no two threads are using the same LogWriter (assuming nothing was using e.g. thread-local storage, which might stipulate that thread assignments would need to be made at initialization and, once made, could never change). So, I suppose it might be good to spawn a finite number of Bro worker threads and assign LogWriters to those (rather than spawning one bro worker thread per logwriter), but I don't see a way around binding an entire logwriter to a thread. What I don't like about this is that it means any scheduling the task manager does would be coarse-grained at best. >> Kind of. Let me throw together a diagram and send it out; it's >> something that should probably end up in the documentation anyway :) >> > > Sounds good. This stuff is much better to comprehend with a visual aid! > Rough cut of a general overview is attached. Probably going to change quite a bit, but I think it might help illustrate the way the system works overall. The architecture, at the moment, is geared toward supporting transparent replacement of threaded log writers with writers that work across sockets and / or IPC; anything the implements QueueInterface can be used to pass messages back and forth. Before this would actually work in a network / IPC context, though, the messages would need to support serialization; haven't done that yet because it hasn't really been a priority. Feedback / corrections / "THAT'S A STUPID MODEL!" are always appreciated. --Gilbert -------------- next part -------------- A non-text attachment was scrubbed... Name: LogSystemOverview.pdf.bz2 Type: application/x-bzip Size: 897244 bytes Desc: not available Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20110728/4403b3d7/attachment-0001.bin From seth at icir.org Fri Jul 29 04:30:54 2011 From: seth at icir.org (Seth Hall) Date: Fri, 29 Jul 2011 07:30:54 -0400 Subject: [Bro-Dev] Tunnels In-Reply-To: <4E320B8A.8060305@icir.org> References: <4E31CB3F.3090400@icir.org> <9E5DC6EA-AB5A-4B10-A2B7-FE999D891EC8@icir.org> <4E320B8A.8060305@icir.org> Message-ID: On Jul 28, 2011, at 9:23 PM, Gregor Maier wrote: > * removing the parent headers completely but generate an event to > associate the child connection with the parent connection (1) That would probably work well once we figured out how to properly deal with (1). :) > (1) not that it's not necessarily a connection. 6to4 and 6in4 are directly on top of IP (using proto 41). So Bro wouldn't even see these packets because it only handles TCP/UDP/ICMP (you would get a weird though) Almost makes me wonder if eventually we'd want to have fake IP connections similarly to the fake udp connections? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Fri Jul 29 08:32:21 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 29 Jul 2011 15:32:21 -0000 Subject: [Bro-Dev] #512: Make the mask_addr function return a subnet In-Reply-To: <046.90c2b99352e56268367772022621b3e4@tracker.bro-ids.org> References: <046.90c2b99352e56268367772022621b3e4@tracker.bro-ids.org> Message-ID: <061.723bf447322cacfe6272c04d47012cef@tracker.bro-ids.org> #512: Make the mask_addr function return a subnet -----------------------------+--------------------- Reporter: seth | Owner: Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: preview -----------------------------+--------------------- Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From vern at icir.org Fri Jul 29 10:13:50 2011 From: vern at icir.org (Vern Paxson) Date: Fri, 29 Jul 2011 10:13:50 -0700 Subject: [Bro-Dev] Tunnels In-Reply-To: (Fri, 29 Jul 2011 07:30:54 EDT). Message-ID: <20110729171350.DB9B4322C99@taffy.ICSI.Berkeley.EDU> > Almost makes me wonder if eventually we'd want to have fake IP connections similarly to the fake udp connections? I'm not following this. Seems we'd instead want (1) a one-time event that identifies the presence of a tunnel, (2) regular processing (via an analyzer chain) of the traffic inside the tunnel, and (3) a way to tell that a give connection record (or other network event) ultimately stems from tunneled traffic. Vern From seth at icir.org Fri Jul 29 10:20:43 2011 From: seth at icir.org (Seth Hall) Date: Fri, 29 Jul 2011 13:20:43 -0400 Subject: [Bro-Dev] Tunnels In-Reply-To: <20110729171350.DB9B4322C99@taffy.ICSI.Berkeley.EDU> References: <20110729171350.DB9B4322C99@taffy.ICSI.Berkeley.EDU> Message-ID: <8F138F08-CCD8-475E-A0F7-D11B6282DBCB@icir.org> On Jul 29, 2011, at 1:13 PM, Vern Paxson wrote: > I'm not following this. Seems we'd instead want (1) a one-time event > that identifies the presence of a tunnel, (2) regular processing (via > an analyzer chain) of the traffic inside the tunnel, and (3) a way to > tell that a give connection record (or other network event) ultimately > stems from tunneled traffic. Yep, your way's much simpler. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From robin at icir.org Fri Jul 29 11:19:00 2011 From: robin at icir.org (Robin Sommer) Date: Fri, 29 Jul 2011 11:19:00 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/trace-summary] fastpath: Teach trace-summary to parse Bro's new conn.log format (closes #520) (975204a) In-Reply-To: <201107281551.p6SFpauH011943@bro-ids.icir.org> References: <201107281551.p6SFpauH011943@bro-ids.icir.org> Message-ID: <20110729181900.GK3901@icir.org> On Thu, Jul 28, 2011 at 08:51 -0700, Jonathan Siwek wrote: > Teach trace-summary to parse Bro's new conn.log format (closes #520) Thanks! But what would you think about keeping compatibility for the old format as well? That might helpful for folks using trace-summary on their own and potentiallu with old logs. By default, trace-summary could try to figure out which format it is (like by looking for the new "#" header), but then also have a switch to set it explicitly. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Fri Jul 29 11:32:24 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 29 Jul 2011 18:32:24 -0000 Subject: [Bro-Dev] #520: trace-summary can't parse new conn.log In-Reply-To: <048.1a6560990d4e6ee8693add3a3cac1151@tracker.bro-ids.org> References: <048.1a6560990d4e6ee8693add3a3cac1151@tracker.bro-ids.org> Message-ID: <063.e3ea246e6fffc552e120dcd4c5fda6f1@tracker.bro-ids.org> #520: trace-summary can't parse new conn.log ----------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Problem | Status: reopened Priority: Normal | Milestone: Bro1.6 Component: trace-summary | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by jsiwek): * status: closed => reopened * resolution: fixed => Comment: >Thanks! But what would you think about keeping compatibility for the >old format as well? That might helpful for folks using trace-summary >on their own and potentiallu with old logs. > >By default, trace-summary could try to figure out which format it is >(like by looking for the new "#" header), but then also have a switch >to set it explicitly. > >Robin ok -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Jul 29 13:04:44 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 29 Jul 2011 20:04:44 -0000 Subject: [Bro-Dev] #509: SMTP rework In-Reply-To: <046.35777c113d149585390ec44632cce689@tracker.bro-ids.org> References: <046.35777c113d149585390ec44632cce689@tracker.bro-ids.org> Message-ID: <061.2d3f4aec32537d077c544ea4626c668a@tracker.bro-ids.org> #509: SMTP rework ---------------------+------------------------ Reporter: seth | Owner: jsiwek Type: Task | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: preview ---------------------+------------------------ Comment (by jsiwek): In [17d74c23dbb60bcf2917d3f1b8ce2d8dc46adbd2/bro]: {{{ #!CommitTicketReference repository="bro" revision="17d74c23dbb60bcf2917d3f1b8ce2d8dc46adbd2" SMTP script refactor. (addresses #509) - message header state tracking is now done by handling mime_one_header instead of parsing the data in the smtp_data event - changed the logging point to be when an smtp_reply is seen in response to the end of a DATA section - the smtp package now uses it's own mime script and logging stream for logging entities, extraction, etc. - fixes for mime file extraction: now logs the extracted file name, and the count of extracted files needed to be maintained in the State record }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Fri Jul 29 13:52:37 2011 From: robin at icir.org (Robin Sommer) Date: Fri, 29 Jul 2011 13:52:37 -0700 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <4E32152C.1000407@ohio.edu> References: <4E163B71.6050804@ohio.edu> <20110708132228.GA84600@icir.org> <1310914521-sup-1714@samurai.icir.org> <20110722042053.GB32586@icir.org> <1311788385-sup-6149@samurai.icir.org> <4E308532.9090205@ohio.edu> <1311840515-sup-4852@samurai.icir.org> <4E31A172.6010600@ohio.edu> <1311878972-sup-3675@samurai.icir.org> <4E32152C.1000407@ohio.edu> Message-ID: <20110729205237.GC12120@icir.org> On Thu, Jul 28, 2011 at 19:04 -0700, you wrote: > So, I suppose it might be good to spawn a finite number of Bro worker > threads and assign LogWriters to those Gilbert and I talked about this a bit more, but for the record: while generally there's nothing wrong with that, I'm not sure we actually need it here. The writer threads are long-lived and many of them won't even have much load. My guess is that the OS will actually be doing just fine scheduling them for us. So my suggestion is to go with individual threads for now and see if we run into problems. If we really do, it should still be quite straight-forward to later switch to a worker thread model. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Fri Jul 29 13:55:21 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 29 Jul 2011 20:55:21 -0000 Subject: [Bro-Dev] #525: topic/jsiwek/unit-tests Message-ID: <048.e688faa418a8ee4f74f7ed57dd6e3426@tracker.bro-ids.org> #525: topic/jsiwek/unit-tests ---------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ This branch currently has unit tests for all the scripts in policy/utils. It also has fixes for some of them: paths.bro - rename extract_directory() to extract_path() (later seemed clearer) and made it work with more than just path string in FTP response msgs - rename build_full_path() and absolute_path() - compress_path() should now work with relative paths also directions-and-hosts.bro - check both connection endpoints when determining direction respectful of local network. addrs.bro - fixed the TODO about making check for valid IPv6 string formats more robust. site.bro - fix error in find_all_emails() about accessing non-existent table indices. I might ask to periodically merge this branch in the future if the task of creating tests for pre-existing scripts keeps getting drawn out. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Jul 29 15:43:35 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 29 Jul 2011 22:43:35 -0000 Subject: [Bro-Dev] #526: topic/jsiwek/dynamic-example-install Message-ID: <048.f9efdb491c054854e8b30d6526373a16@tracker.bro-ids.org> #526: topic/jsiwek/dynamic-example-install ---------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ This branch exists in `bro`, `broccoli`, and `broctl` repos and changes example config files to only be installed when there exists a version of the config file on disk already and it differs from the version about to installed from the distribution. Note that the current master version of `broctl` still attempts to install a "local.bro" while the `bro` repo is now managing the installation of that file, resulting in an example file being installed when that repo combination is used. The `policy-scripts-new` branch in `broctl` won't be doing that. -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Sat Jul 30 13:30:28 2011 From: gregor at icir.org (Gregor Maier) Date: Sat, 30 Jul 2011 13:30:28 -0700 Subject: [Bro-Dev] Directly committing to master Message-ID: <4E3469E4.5010304@icir.org> Hi, recently there have been a bunch of commits directly on master. Is there a particular reason for that? I would really prefer it if we could stick to our development process where work gets committed to topic branches and the topic branches are then merged into master. (These merges can happen often. I wouldn't mind that) cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From vallentin at icir.org Sat Jul 30 15:56:41 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Sat, 30 Jul 2011 15:56:41 -0700 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <4E32152C.1000407@ohio.edu> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> <20110707215133.GF77684@icir.org> <4E163B71.6050804@ohio.edu> <20110708132228.GA84600@icir.org> <1310914521-sup-1714@samurai.icir.org> <20110722042053.GB32586@icir.org> <1311788385-sup-6149@samurai.icir.org> <4E308532.9090205@ohio.edu> <1311840515-sup-4852@samurai.icir.org> <4E31A172.6010600@ohio.edu> <1311878972-sup-3675@samurai.icir.org> <4E32152C.1000407@ohio.edu> Message-ID: <1312066147-sup-5217@samurai.icir.org> (I read Robin's follow up and have no objections with keeping the hardware thread model instead of a task-based worker thread model. My comments below are just for the record.) > [..] this brings up a more general problem: what if we use a library > that doesn't offer us the ability to do this? That's a big issue. Some research at the Berkeley ParLab tried to address this by providing a broker library; sort of a meta scheduler that seeks to make sure cross-library calls do not hurt concurrency. Alas, I cannot remember the name of the project but if you are interested I can find it out. > Then again, if the library doesn't let us do this, maybe we just > shouldn't be using that particular library...? If the library is uncontrollable with respect to the number of threads it spawns, that's tough. Though most well-written libraries (not stand-alone applications) I came across have some way to configure or limit the number of used threads. Even if we used 4 different libraries, each of which can be trimmed down to two threads, I'd say we have a tractable situation because the number of cores will probably grow faster than the minimum number of threads in libraries we use. > What I don't like about this is that it means any scheduling the task > manager does would be coarse-grained at best. Agreed, the logging framework is not a component of Bro that would reap major benefits from the worker thread model. The reason why I proposed the task-based architecture is that it is a strong spine for modern concurrent applications. However, Bro has enough muscle at this point, which is why focusing on other body parts is probably a better idea. (I hope this metaphor is not too awkward :-) > Rough cut of a general overview is attached. Probably going to change > quite a bit, but I think it might help illustrate the way the system > works overall. Cool, that's a nice diagram. Having similar ones for the big parts of Bro is a valuable resource for developers. Some quick questions/comments: - In the LogMgr description, what's a log stream object? - I'm not sure if I understand the crossed-out CONTEXT BOUNDARY text. Does this essentially mean that the queue interface can (in the future) adapt based on how the distributed system is configured? I.e., if reader and writer are part of the same program, it is some sort of IN_PROC queue, and for IPC and networking it will transparently serialize the messages? - What is the Pull FIFO for? Does the LogEmissary receive feedback from the LogWriter? - Just out of curiousity, what type of thread-safe queue is it? Single-writer/single-reader? Single-writer/multiple-reader? There are zillions of thread-safe queue implementations out there, each optimized for some different use case! While we're at it, is the LogEmissary to LogWriter multiplicity 1:1 or 1:n? - About BasicThread: it seems there is one channel for both control and data messages, i.e., the queue can include a special terminate event to shutdown the BasicThread. Are these control messages filtered out in the base class (BasicThread) code before the message is passed to the child (E.g. LogWriter) code? > Feedback / corrections / "THAT'S A STUPID MODEL!" are always > appreciated. I like the architecture (+1 with your words ;-) and am looking forward to see it in action. What I am really interested in are profiling benchmarks (very easy with Google perftools) to see where the non-I/O bottlenecks occur. Sorry if I have missed it, but do we use now 0mq as messaging middle-layer or is all message passing based on custom code? Matthias From bro at tracker.bro-ids.org Sat Jul 30 19:52:18 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 31 Jul 2011 02:52:18 -0000 Subject: [Bro-Dev] #527: --with-perftools broken? Message-ID: <046.8cf6b9c3acfe750a6f4cb75681b3b0ed@tracker.bro-ids.org> #527: --with-perftools broken? ---------------------+------------------------ Reporter: seth | Owner: jsiwek Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ When I gave the configure script the --with-perftools option, I got an error message. {{{ CMake Warning: The variable, 'GooglePerftools_ROOT_DIR', specified manually, was not used during the generation. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Sat Jul 30 20:03:22 2011 From: seth at icir.org (Seth Hall) Date: Sat, 30 Jul 2011 23:03:22 -0400 Subject: [Bro-Dev] Directly committing to master In-Reply-To: <4E3469E4.5010304@icir.org> References: <4E3469E4.5010304@icir.org> Message-ID: On Jul 30, 2011, at 4:30 PM, Gregor Maier wrote: > recently there have been a bunch of commits directly on master. Is there > a particular reason for that? I would really prefer it if we could stick > to our development process where work gets committed to topic branches > and the topic branches are then merged into master. (These merges can > happen often. I wouldn't mind that) We had agreed that script development would happen directly on master temporarily but I can definitely see that causing trouble. I think I agree and I'll move that development over to a branch. The SSL fix was different. It was a very simple change and the absence of the fix was causing trouble for a few people. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From slagell at ncsa.illinois.edu Sat Jul 30 20:06:24 2011 From: slagell at ncsa.illinois.edu (Adam J. Slagell) Date: Sat, 30 Jul 2011 22:06:24 -0500 Subject: [Bro-Dev] Directly committing to master In-Reply-To: References: <4E3469E4.5010304@icir.org> Message-ID: On Jul 30, 2011, at 10:03 PM, Seth Hall wrote: > > On Jul 30, 2011, at 4:30 PM, Gregor Maier wrote: > >> recently there have been a bunch of commits directly on master. Is there >> a particular reason for that? I would really prefer it if we could stick >> to our development process where work gets committed to topic branches >> and the topic branches are then merged into master. (These merges can >> happen often. I wouldn't mind that) > > > We had agreed that script development would happen directly on master temporarily but I can definitely see that causing trouble. I think I agree and I'll move that development over to a branch. > > The SSL fix was different. It was a very simple change and the absence of the fix was causing trouble for a few people. Still, isn't that kind of what fastpath is for? ------ Adam J. Slagell, CISSP Sr. Security Engineer National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info 217.244.8965 From seth at icir.org Sat Jul 30 20:18:33 2011 From: seth at icir.org (Seth Hall) Date: Sat, 30 Jul 2011 23:18:33 -0400 Subject: [Bro-Dev] Directly committing to master In-Reply-To: References: <4E3469E4.5010304@icir.org> Message-ID: On Jul 30, 2011, at 11:06 PM, Adam J. Slagell wrote: > Still, isn't that kind of what fastpath is for? Argh, good point. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From robin at icir.org Sat Jul 30 20:49:33 2011 From: robin at icir.org (Robin Sommer) Date: Sat, 30 Jul 2011 20:49:33 -0700 Subject: [Bro-Dev] Directly committing to master In-Reply-To: References: <4E3469E4.5010304@icir.org> Message-ID: <20110731034933.GA24247@icir.org> Gregor, what's the original problem you ran into? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gc355804 at ohio.edu Sat Jul 30 22:30:27 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Sun, 31 Jul 2011 01:30:27 -0400 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <1312066147-sup-5217@samurai.icir.org> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> <20110707215133.GF77684@icir.org> <4E163B71.6050804@ohio.edu> <20110708132228.GA84600@icir.org> <1310914521-sup-1714@samurai.icir.org> <20110722042053.GB32586@icir.org> <1311788385-sup-6149@samurai.icir.org> <4E308532.9090205@ohio.edu> <1311840515-sup-4852@samurai.icir.org> <4E31A172.6010600@ohio.edu> <1311878972-sup-3675@samurai.icir.org> <4E32152C.1000407@ohio.edu> <1312066147-sup-5217@samurai.icir.org> Message-ID: <4E34E873.70109@ohio.edu> On 7/30/2011 6:56 PM, Matthias Vallentin wrote: > (I read Robin's follow up and have no objections with keeping the > hardware thread model instead of a task-based worker thread model. > My comments below are just for the record.) > >> [..] this brings up a more general problem: what if we use a library >> that doesn't offer us the ability to do this? > That's a big issue. Some research at the Berkeley ParLab tried to > address this by providing a broker library; sort of a meta scheduler > that seeks to make sure cross-library calls do not hurt concurrency. > Alas, I cannot remember the name of the project but if you are > interested I can find it out. > Sure, if it wouldn't be too much trouble :) I'd love to see how they manage to do that. > Cool, that's a nice diagram. Thanks :) > Having similar ones for the big parts of > Bro is a valuable resource for developers. Some quick > questions/comments: > > - In the LogMgr description, what's a log stream object? Oh, yeah. That probably needs to go in there. To answer the question: a bro log stream is defined by three things -- a path (where do I write), a record type (what do I write), and a writer type (how do I write). The stream object holds a little bit of state ("am I pushing my logs through a remote serializer?" "is this log stream enabled or disabled?" etc). Also, since the types used by the log writers differ somewhat from the types used by bro internally, the stream holds some information used to help with that mapping. That said, stream objects could (and probably eventually should) be rolled into the LogEmissary type. > - I'm not sure if I understand the crossed-out CONTEXT BOUNDARY > text. Does this essentially mean that the queue interface can (in > the future) adapt based on how the distributed system is > configured? I.e., if reader and writer are part of the same > program, it is some sort of IN_PROC queue, and for IPC and > networking it will transparently serialize the messages? Yeah, that was the intent. Wasn't really sure quite how to illustrate that; any thoughts on a better way? > - What is the Pull FIFO for? Does the LogEmissary receive > feedback from the LogWriter? Yes. Currently, that feedback is limited to error messages, but there are other messages planned as well. > - Just out of curiousity, what type of thread-safe queue is it? > Single-writer/single-reader? Single-writer/multiple-reader? There > are zillions of thread-safe queue implementations out there, each > optimized for some different use case! I don't know if I'd really call this one "optimized" :) The queue was thrown together pretty quickly; it's largely targeted for single producer / single consumer, but I believe multiple producer / single consumer should work as well. Trying to use multiple consumers with this queue would likely result in some kind of universe-ending quantum event involving the LHC, the deflector dish on the USS Enterprise, and a relatively cute kitten with gray fur and black tiger stripes. Alternatively, the program could just crash and / or deadlock, but that's not nearly as much fun to contemplate. Regardless, verifying the above and documenting the properties of said queue would probably be an excellent idea. Thanks :) > While we're at it, is the > LogEmissary to LogWriter multiplicity 1:1 or 1:n? > 1:1; it wouldn't be difficult to make it otherwise, but I haven't been able to think of a good reason to do so. > - About BasicThread: it seems there is one channel for both control > and data messages, i.e., the queue can include a special terminate > event to shutdown the BasicThread. Are these control messages > filtered out in the base class (BasicThread) code before the > message is passed to the child (E.g. LogWriter) code? > More or less. An additional piece of the message (not shown on the diagram. . . not quite sure how to illustrate this yet) involves a reference to the LogWriter which is used to process the message. For example: class FlushMessage : public EventMessage { public: FlushMessage(LogWriter &ref) : _ref(ref) { } void process() { ref.flush(); } private: LogWriter &_ref; }; For the purposes of serialization, the reference can be expressed as a pair. Then, when BasicThread pulls the event off of the queue and calls process(), the appropriate LogWriter function is called. For a terminate message, then: class TerminateThread : public MessageEvent { public: TerminateThread(ThreadInterface &ref) : ref(ref) { } bool process() { ref.stop(); return true; } private: ThreadInterface &ref; }; where BasicThread inherits from ThreadInterface, and thus the BasicThread is stopped when the event is processed. I'm always open to better ideas :) >> Feedback / corrections / "THAT'S A STUPID MODEL!" are always >> appreciated. > I like the architecture (+1 with your words ;-) and am looking forward > to see it in action. What I am really interested in are profiling > benchmarks (very easy with Google perftools) to see where the non-I/O > bottlenecks occur. Probably going to start working on that after a bit; there's a crossover cable hooked up between two development machines (thanks Robin!) and a few GB of traces ready to go, but took a break to address testing and tool-related stuff. If you'd happen to know of a good way to measure contention on a Linux system, I'd love to hear about it; I'm planning on writing a few stap scripts to help out here, but it'd save me a lot of time if there were something that existed already and seemed to work pretty well. > Sorry if I have missed it, but do we use now 0mq as messaging > middle-layer or is all message passing based on custom code? All custom code. It didn't seem to make sense to require 0mq as a dependency when the logging infrastructure was the only thing that would use it. --Gilbert From vallentin at icir.org Sun Jul 31 05:27:38 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Sun, 31 Jul 2011 05:27:38 -0700 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <4E34E873.70109@ohio.edu> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> <20110707215133.GF77684@icir.org> <4E163B71.6050804@ohio.edu> <20110708132228.GA84600@icir.org> <1310914521-sup-1714@samurai.icir.org> <20110722042053.GB32586@icir.org> <1311788385-sup-6149@samurai.icir.org> <4E308532.9090205@ohio.edu> <1311840515-sup-4852@samurai.icir.org> <4E31A172.6010600@ohio.edu> <1311878972-sup-3675@samurai.icir.org> <4E32152C.1000407@ohio.edu> <1312066147-sup-5217@samurai.icir.org> <4E34E873.70109@ohio.edu> Message-ID: <1312113024-sup-7059@samurai.icir.org> > Sure, if it wouldn't be too much trouble :) I'd love to see how they > manage to do that. The project is called Lithe and has it's own web presence here: http://parlab.eecs.berkeley.edu/lithe I haven't looked into yet, but the linked paper is probably the best way to absorb the details. > To answer the question: a bro log stream is defined by three things -- a > path (where do I write), a record type (what do I write), and a writer > type (how do I write). Got it. > Yeah, that was the intent. Wasn't really sure quite how to illustrate > that; any thoughts on a better way? Not really. Maybe several boxes inside the QueueInterface to illustrate that "there is something going on" inside this intelligent queue?! > The queue was thrown together pretty quickly; it's largely targeted > for single producer / single consumer, but I believe multiple producer > / single consumer should work as well. If it works, it works. I was just curious about the implementation. > Trying to use multiple consumers with this queue would likely result in > some kind of universe-ending quantum event involving the LHC, the > deflector dish on the USS Enterprise, and a relatively cute kitten with > gray fur and black tiger stripes. I see... the kitten could cause real trouble [1]. > If you'd happen to know of a good way to measure contention on a Linux > system, I'd love to hear about it; I've only used Google perftools thus far to profile CPU and heap, but haven't come across something similar for I/O. Matthias [1] http://www.psiopradio.com/wp-content/uploads/sniper_kitten1.jpg From seth at icir.org Sun Jul 31 18:00:00 2011 From: seth at icir.org (Seth Hall) Date: Sun, 31 Jul 2011 21:00:00 -0400 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <4E34E873.70109@ohio.edu> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> <20110707215133.GF77684@icir.org> <4E163B71.6050804@ohio.edu> <20110708132228.GA84600@icir.org> <1310914521-sup-1714@samurai.icir.org> <20110722042053.GB32586@icir.org> <1311788385-sup-6149@samurai.icir.org> <4E308532.9090205@ohio.edu> <1311840515-sup-4852@samurai.icir.org> <4E31A172.6010600@ohio.edu> <1311878972-sup-3675@samurai.icir.org> <4E32152C.1000407@ohio.edu> <1312066147-sup-5217@samurai.icir.org> <4E34E873.70109@ohio.edu> Message-ID: <05B3721C-8B12-4CA3-83DA-F97C8AD0E0AF@icir.org> On Jul 31, 2011, at 1:30 AM, Gilbert Clark wrote: > To answer the question: a bro log stream is defined by three things -- a > path (where do I write), a record type (what do I write), and a writer > type (how do I write). Hm, are logging streams defined slightly differently in the "core" than they are in the scripting language? Streams are only defined by their record type. Where to write and how to write are both handled by filters. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From gc355804 at ohio.edu Sun Jul 31 20:47:01 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Sun, 31 Jul 2011 23:47:01 -0400 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <05B3721C-8B12-4CA3-83DA-F97C8AD0E0AF@icir.org> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> <20110707215133.GF77684@icir.org> <4E163B71.6050804@ohio.edu> <20110708132228.GA84600@icir.org> <1310914521-sup-1714@samurai.icir.org> <20110722042053.GB32586@icir.org> <1311788385-sup-6149@samurai.icir.org> <4E308532.9090205@ohio.edu> <1311840515-sup-4852@samurai.icir.org> <4E31A172.6010600@ohio.edu> <1311878972-sup-3675@samurai.icir.org> <4E32152C.1000407@ohio.edu> <1312066147-sup-5217@samurai.icir.org> <4E34E873.70109@ohio.edu> <05B3721C-8B12-4CA3-83DA-F97C8AD0E0AF@icir.org> Message-ID: <4E3621B5.8010208@ohio.edu> > Hm, are logging streams defined slightly differently in the "core" than they are in the scripting language? Streams are only defined by their record type. Where to write and how to write are both handled by filters. > Internally, filters are assigned to streams (e.g. seem to be part of a stream). They *are* discrete components, though, so maybe it's not a good idea to lump the two together like I did? --Gilbert From bro at tracker.bro-ids.org Sun Jul 31 21:31:07 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 01 Aug 2011 04:31:07 -0000 Subject: [Bro-Dev] #525: topic/jsiwek/unit-tests In-Reply-To: <048.e688faa418a8ee4f74f7ed57dd6e3426@tracker.bro-ids.org> References: <048.e688faa418a8ee4f74f7ed57dd6e3426@tracker.bro-ids.org> Message-ID: <063.6e67f0b6c059a9d0140b7c45664b1cad@tracker.bro-ids.org> #525: topic/jsiwek/unit-tests ----------------------------+------------------------ Reporter: jsiwek | Owner: robin Type: Merge Request | Status: closed Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------------+------------------------ Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [47d9b592637536128c0f0c0163efef6156566efc/bro]: {{{ #!CommitTicketReference repository="bro" revision="47d9b592637536128c0f0c0163efef6156566efc" Merge remote-tracking branch 'origin/topic/jsiwek/unit-tests' * origin/topic/jsiwek/unit-tests: Fix utils/conn-ids test due to renamed conn-ids.bro Moving the test for site.bro to live w/ other utils/ tests. Fix test due to moving of site.bro More policy/utils unit tests and documentation. Updating documentation for some utils/ policy scripts Add unit tests for utils/paths.bro with some changes Adding unit tests for utils. Adding test for utils/addrs.bro. Add unit test for site.bro. Conflicts: policy/utils/site.bro Closes #525. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Jul 31 21:31:07 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 01 Aug 2011 04:31:07 -0000 Subject: [Bro-Dev] #526: topic/jsiwek/dynamic-example-install In-Reply-To: <048.f9efdb491c054854e8b30d6526373a16@tracker.bro-ids.org> References: <048.f9efdb491c054854e8b30d6526373a16@tracker.bro-ids.org> Message-ID: <063.bfae1551913638292ac4367de0e7e7c0@tracker.bro-ids.org> #526: topic/jsiwek/dynamic-example-install ----------------------------+------------------------ Reporter: jsiwek | Owner: robin Type: Merge Request | Status: closed Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------------+------------------------ Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [a5031ccce1f7fb756e034a41f30c6ffb8bf8c7df/bro]: {{{ #!CommitTicketReference repository="bro" revision="a5031ccce1f7fb756e034a41f30c6ffb8bf8c7df" Merge remote-tracking branch 'origin/topic/jsiwek/dynamic-example-install' * origin/topic/jsiwek/dynamic-example-install: Install example config files dynamically. Closes #526. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker