[Bro] Traffic Volume Calculation Using Bro's Connection Log

[Naveed Anwar]

Hi Seth/John,

Thank you for your responses. I have a follow up question.

Here's a quick recap of what I need to do: I want to use Bro to calculate
the total volume of traffic captured in a pcap file, including all headers
up to (and including) Ethernet headers.

Following your suggestion to sum the resp_ip_bytes and orig_ip_bytes
columns of the
conn.log generated over the trace, I now use this script to calculate the
volume:

cat conn.log | awk 'BEGIN{FS="\t"; count=0;} {count=count+$17; count+=$19}
END {print count}'

I've tried running this over conn logs generated from two different pcap
files. In both cases, I get a count that is smaller than the size of the
pcap file. Which is fine, because like you said, ethernet headers and pcap
headers are still not included.  The problem is that when I ran this script
on the conn.log generated over a 500 GB trace, the output was 376 GB. If I
calculate the total Ethernet header size (assuming 14 bytes per packet) AND
the total pcap header size (assuming 16 bytes per packet) for that trace,
it comes to around 21 GB. That means (500-376-20 =) ~ 104 GB is still
unaccounted for. I'm trying to understand why that would be.

Perhaps that is because of packets with unknown transport protocols? You
said that packet types that are not supported will also not be included in
the byte count. By unsupported packet types, are you referring to
connections for which the value of the enum "proto" is "unknown_protocol"?
If yes, I've seen that conn.log is showing ONLY TCP, UDP, and ICMP in the
"proto" field for my 500 GB trace. Does this mean that when the protocol is
unknown, the record is not included in conn.log at all? Because that's the
only explanation I can think of for the unaccounted 104 GB!

Can you please comment, and also tell me if I'm doing something incorrect,
and if so, how I should be calculating the volume instead?

Thankyou
Zainab


On Thu, Nov 7, 2013 at 1:48 PM, Naveed Anwar <hunarame at gmail.com> wrote:

> Hi,
>
> I'm facing a small problem when running Bro. I'm trying to calculate the
> volume of traffic generated per host. I have a set of pcap files, each
> containing traffic from a single host. I thought I could run Bro on each
> pcap file, and then sum the orig_bytes and resp_bytes columns in conn.log
> to get the total volume of traffic for one host. However when I run Bro on
> a 250 MB pcap file, the sum of these two columns  is only 107 MB
> approximately, and not 250 MB as I expected. Is there any alternate method
> for calculating the volume of traffic generated by one host?
>
> Here's the script I ran to get the sum:
> cat conn.log | awk 'BEGIN{FS="\t"; count=0;} {count=count+$10; count+=$11}
> END {print count;}'
>
> This was the output of the script (which I expected would be 250 MB
> instead):
> 107790112 bytes
>
> It would be great if you could help me resolve this issue!
>
> Thank you,
> Zainab
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131121/889cf4fb/attachment.html