[Bro] Writing logs to both ACII and JSON

[James Gordon]

Jan, + re-adding the bro mailing list because email is hard and I
accidentally removed it -  and in case there's a bug impacting this script
in v 2.5,

I tested this script on my physical security onion box, as well a security
onion VM and a CentOS VM both with fresh installs of Bro 2.5. I tested with
live network traffic and with a pcap and consistently get different results
in my JSON log dir every time I run bro against the pcap.

When I run bro against a pcap, I get the following error:
"expression error in /opt/bro/share/bro/test/./add-json.bro, line 34: field
value missing [Log::filter$path]"

It looks like that line refers back to the json path. I have the json path
defined as: const path_json = "/nsm/bro/logs/json/" &redef; - is this the
correct way to define the log path?

Here's some examples of the inconsistencies I see (this is reproduceable on
all three systems). I'll run the same pcap through Bro twice and we'll get
a different number of JSON logs, and different entries in the files - but
ASCII logs always turn out the same.

root at sensor:/home/sensor/test# /opt/bro/bin/bro -r test.pcap
/opt/bro/share/bro/site/local.bro
expression error in /opt/bro/share/bro/test/./add-json.bro, line 34: field
value missing [Log::filter$path]
root at sensor:/home/sensor/test# ls
capture_loss.log  dhcp.log  files.log  loaded_scripts.log
 packet_filter.log  ssl.log    test.pcap   weird.log
conn.log          dns.log   http.log   notice.log          reporter.log
  stats.log  tunnel.log  x509.log
root at sensor:/home/sensor/test# ls | wc -l
16
root at sensor:/home/sensor/test# cat conn.log | wc -l
1631

root at sensor:/home/sensor/test# ls /nsm/bro/logs/json/
dhcp-json.log  tunnel-json.log  x509-json.log
root at sensor:/home/sensor/test# ls /nsm/bro/logs/json/ | wc -l
3

As you can see there was no JSON conn log generated - so i'll compare the
dhcp logs:

root at sensor:/home/sensor/test# cat dhcp.log | wc -l
11
root at sensor:/home/sensor/test# cat /nsm/bro/logs/json/dhcp-json.log | wc -l
2

Some of the lines (8) in the ASCII file are headers so this log only missed
one entry. It still missed logging all 1631 connections in the pcap to
conn.log. I'll clear out the logs now and try again, and we'll get a
different number types of json logs created.


root at sensor:/home/sensor/test# rm *.log
root at sensor:/home/sensor/test# rm /nsm/bro/logs/json/*

root at sensor:/home/sensor/test# /opt/bro/bin/bro -r test.pcap
/opt/bro/share/bro/site/local.bro
expression error in /opt/bro/share/bro/test/./add-json.bro, line 34: field
value missing [Log::filter$path]
root at sensor:/home/sensor/test# ls
capture_loss.log  dhcp.log  files.log  loaded_scripts.log
 packet_filter.log  ssl.log    test.pcap   weird.log
conn.log          dns.log   http.log   notice.log          reporter.log
  stats.log  tunnel.log  x509.log
root at sensor:/home/sensor/test# ls | wc -l
16
root at sensor:/home/sensor/test# cat conn.log  | wc -l
1631
root at sensor:/home/sensor/test# ls /nsm/bro/logs/json/
capture_loss-json.log  files-json.log           packet_filter-json.log
 weird-json.log
conn-json.log          loaded_scripts-json.log  reporter-json.log
x509-json.log
dhcp-json.log          notice-json.log          tunnel-json.log
root at sensor:/home/sensor/test# ls /nsm/bro/logs/json/ | wc -l
11
root at sensor:/home/sensor/test# cat /nsm/bro/logs/json/conn-json.log | wc -l
1622

This time it logged all the connections, but it failed to even create http,
ssl, stats, or dns json logs.

This script is exactly the functionality I need, I just can't seem to get
it working correctly. I don't begin to understand why I get different
results every time I run the same pcap through Bro.

Thanks!


James Gordon






On Tue, Jan 10, 2017 at 2:21 AM, Jan Grashöfer <jan.grashoefer at gmail.com>
wrote:

> > The logs don't ever seem to equal out. I noticed this first with the
> tunnel
> > log because tunnel.log was present in the ascii log dir, but not in the
> > json log dir. However the issue seems consistent across all my log files
> -
> > some events just aren't making it to the json files.
>
> In this case I would try to reproduce this issue consistently. You could
> first try it by reading a pcap into Bro. I doubt that will cause
> different ASCII and JSON logs but it might be worth a try. If that does
> work, you might want to replay traffic into your setup. You could try
> enable only JSON, only ASCII and both and see which ones differ.
>
> Looking at the script I linked, there is no logical explanation for this
> behavior in the code. Therefore I guess its a deployment related issue
> or maybe even some kind of bug. Please, let me know about your findings.
>
> Cheers,
> Jan
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170111/32193168/attachment.html