[Bro] Differences in processing multiple traces with BRO and ipsumdump

[Veronica Estrada]

Hi everyone,

I am puzzled about the outcomes of using ipsumdump or BRO for processing
multiple pcap files.

I am using BRO to analyze anomalities in my 12 hours captured network
traffic which was saved in 4 Gb pcap files. I want that BRO consider the
cases when a connection may have been split in two or more files. I was
using ipsumdump to solve this, but I found that some files have errors and
cause ipsumdump to crush with this message:
ToDump(bigPcap1.pcap): Inappropriate ioctl for device

Using the capinfo tool I detected that some of my files have packet size
larger than normal (65535), so using tshark I cut the part of the file with
problems. For example:
capinfos: An error occurred after reading 3830659 packets from
"trace2.pcap": File contains a record that's not valid.
(pcap: File has 4065648712-byte packet, bigger than maximum of 65535)
So I create a reduced version of trace2.pcap with tshark:
/usr/sbin/tshark -c 3830659 -r trace2.pcap -w trace2-new.pcap

This solution seemed to work fine, all the ***-new.pcap have no errors while
reading with capinfo or wireshark, but even so  there are some that still
cause problems for processing. For example:

I processed the following files in 3 different ways:
trace1.pcap, trace2-new.pcap, trace3.pcap (trace2.pcap was replaced because
of the packet size error)

FIRST TRY - using ipsumdump with collate option:
ipsumdump --collate -w - trace* |bro -r - brolite myenvironment -f "tcp or
udp or icmp" dpd_conn_logs=T dpd detect_protocols dyn_disable irc-bot proxy
ftp

Output> 9.7 MB conn.log with 114861 lines (number of connections)

SECOND TRY - using ipsumpdump without collate option
ipsumdump --collate -w - trace* |bro -r - brolite myenvironment -f "tcp or
udp or icmp" dpd_conn_logs=T dpd detect_protocols dyn_disable irc-bot proxy
ftp

Output:
19 Mbytes conn.log with 228922 lines  with 950 repeated connections

THIRD TRY - without ipsumdump:
/usr/local/bro/bin/bro -r trace1.pcap -r trace2-new.pcap -r trace3.pcap
brolite todai -f "tcp or udp or icmp" dpd_conn_logs=T dpd detect-protocols
dyn-disable irc-bot proxy ftp 2>bro-error3.log

Output:
15 Mbytes conn.log with 169168 lines, connections are not repeated

COMMENTS:
pcap files has not overlap traffic (it was checked with trace-summary using
first packet seen and last packet seen).
I tried the ipsumdump with both collate and no collate option because when I
used ipsumdump only (without bro), with collate option the resulted larger
pcap file was a 7.9 GB file but without collate option the resulted file was
12.GB (trace1.pcap: 4 MB, trace2-new.pcap: 3.9GB, trace3.pcap: 4GB).
Besides, while using ipsumpdump --collate alone, the progress bar showed
something like this:
66%******************              |8017MB ETAToDump(LargerTrace.pcap):
Success
100%****************************|12113MB
But the progress bar for ipsumdump without the collate option didn't split
and reach the 100% 12113MB.

If anyone can illuminate this matter, it will be a great help.

Veronica
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20100618/f1087dd8/attachment.html