[Bro] http incomplete file extraction (Files::ANALYZER_EXTRACT)

[Frank Meier]

Hi!

On Mo, Mär 30, 2015 at 6:54 , Siwek, Jon <jsiwek at illinois.edu> wrote:

> In files.log, the value of total_bytes is just taken from the HTTP 
> Content-Length header.  Since the value of seen_bytes is less than 
> total_bytes, you can suspect Bro didn’t see the full file for some 
> reason.  Do you have a weird.log containing any obvious clues?  Else, 
> I may need the original pcap to understand what went wrong.

The weird.log states some "above_hole_data_without_any_acks", but why 
does it work with tcpflow?

Here is what I did:

1) I downloaded the test file: wget 
http://ipv4.download.thinkbroadband.com/5MB.zip
2) Gathered the pcap: tcpdump -s0 -i eth0 -w download.pcap port http
3) checked if the file was completely captured with tcpflow:
tcpflow -FT -e http -r download.pcap
md5sums do match: 
~/bro-liste$ md5sum 
2015-04-01T07:45:00Z080.249.099.148.00080-192.168.002.103.42716-HTTPBODY-001.zip
b3215c06647bc550406a9c8ccc378756  
2015-04-01T07:45:00Z080.249.099.148.00080-192.168.002.103.42716-HTTPBODY-001.zip
~/bro-liste$ md5sum 5MB.zip 
b3215c06647bc550406a9c8ccc378756  5MB.zip

4) run bro (revision 32ae94de9ae36060651240a0ee11838e3e572223) with 
simple bro-file:
~/bro-liste$ cat extract.bro 
event file_new(f: fa_file)
{
                Files::add_analyzer(f, Files::ANALYZER_EXTRACT);
}

~/bro-liste$ /usr/local/bro/bin/bro -r download.pcap extract.bro 
1427874309.892545 warning in 
/usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro, line 
54: Your trace file likely has invalid TCP checksums, most likely from 
NIC checksum offloading.

5) Logs from bro and the pcap: (14mb)
http://www.xup.in/dl,19594721/extract.tar.bz2/

Thanks!

Franky


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150401/a7a965d4/attachment.html