[Bro] Question about how to transfer bro data to java

[Clark, Gilbert]

Just a few more thoughts, for what they're worth.  Might be a reason the following are not good ideas: haven't thought through them too much.  Just listing the first few things that come to mind.

One alternative might be to build a plugin to define a bro script method that would serialize the table data into something Java could understand, send it to Java somehow, and then have Java deserialize it on the receiving end.  If you're looking for simplicity, I think JSON would definitely be easy to deserialize into nested HashMaps on the Java end, and would *probably* be pretty straightforward to serialize on the Bro end.  That's not going to be the fastest format to work with, however.

Something that might be even faster (and harder) would be to build a shared memory region, then build a plugin method that would serialize the data into a form that made sense for Java, then bulk copy the table data into that shared memory space.  From there, the Java application could map the shared memory segment into its own space directly, so there wouldn't need to be a deserialization step involved.  The synchronization could be a challenge, though, I think.  Also, pulling tabular data from Java mapped ByteBuffer objects (or equivalent) could be a challenge, depending on the type of data contained in the table.

A slightly more complex twist might be to extend bro's tables to support being backed by shared memory, the idea being that they could be mapped into other processes directly.  I do think this would be *very* hard to get exactly right ... and there would still be some additional overhead in the form of copying / synchronization required to make this kind of IPC work in the first place.  It's hard to say exactly how well (or not) this would perform, I think.

If the Java environment can reasonably run in the bro process space, then another option might be to launch a JVM from within bro.  Write a script extension that would write the table data into a queue, and let a separate Java thread read / process the data being fed to it through said queue.

Note that, last I knew, Java doesn't support shared memory directly ... [1] and [2] might be interesting reading that would be related to this subject and offer practicable workarounds, however.

If the intermediate file option is the most attractive, might consider looking into trying to write to / read from a RAM disk (if that hasn't been done already).  The performance *could* be better ... or not.

Anyway, hope something in there is interesting / useful, and good luck :)

-Gilbert

[1] http://javaforu.blogspot.com/2011/09/offloading-data-from-jvm-heap-little.html
[2] http://javaforu.blogspot.com/2011/09/ram-disk-is-already-in-linux-and-nobody.html

________________________________________
From: bro-bounces at bro.org <bro-bounces at bro.org> on behalf of Seth Hall <seth at icir.org>
Sent: Thursday, December 11, 2014 9:13 AM
To: Ren, Wenyu
Cc: bro at bro.org
Subject: Re: [Bro] Question about how to transfer bro data to java

> On Dec 11, 2014, at 2:14 AM, Ren, Wenyu <wren3 at illinois.edu> wrote:
>
> The way I can come up with is to simply traverse the tree and write all the data in the tree in some certain formats to a text file and read that file in java to construct the structure again from scratches. But this seems extremely inefficient. Do you know any better way to do this?

That’s probably not a bad way to go.  No matter what you do you’re going to end up with Bro serializing and then your java code deserializing all of your data.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/


_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro