[Bro] HTTP Object length calculation

[Ioannis WiCom]

Thank you both for getting back to me.

I understand your point regarding the content-length.

However, it looks to me that there is a contradiction in the calculation. 

When the host receives the full object and it is encoded, bro reports the size of the object in the server (before compression), not the actual bytes in the network (after compression).
When the object is partially downloaded (object_length<content_length) and assuming it was encoded, bro reports the actual bytes in the network (since partial decompression cannot be performed).

Am I missing something?
Is there a way to get the actual bytes transferred?

thanks,
Yannis

> Date: Tue, 13 Sep 2011 13:12:50 -0700
> From: gregor at icir.org
> To: seth at icir.org
> CC: iduckhd at hotmail.com; bro at bro-ids.org
> Subject: Re: [Bro] HTTP Object length calculation
> 
> On 9/13/11 10:05 , Seth Hall wrote:
> >
> > On Sep 13, 2011, at 11:57 AM, Ioannis WiCom wrote:
> >
> >> I have isolated an example TCP connection, and measured the bytes using wireshark. The real object length is equal to the "Content-Length", but the reported by bro is much higher. Therefore, I cannot understand what the value stat$body_length represents.
> >
> > stat$body_length *should* be the actual counted number of bytes that were in the body.  If you see a disparity between the two numbers, the web server could be reporting an incorrect length for the data it's sending.  Could you send the trace file privately?
> 
> Actually that's not exactly the case. Bro reports the body length *after 
> decompression* (for transfer-encodings that use compressions).
> 
> In addition, the Content-Length header is often unreliable. E.g., if an 
> HTTP transfer is interrupted fewer bytes are transferred that reported 
> by Content-Length. This can happen often with (misconfigured) download 
> managers. Or the Content-Length header can also be just plain wrong 
> (HTTP server sends garbage). We did a study with residential traffic and 
> found that the Content-Length header will on average over-report the 
> volume by a factor of about 5 (with some spikes reaching several 100(!))
> 
> cu
> Gregor
> -- 
> Gregor Maier
> <gregor at icir.org>  <gregor at icsi.berkeley.edu>
> Int. Computer Science Institute (ICSI)
> 1947 Center St., Ste. 600
> Berkeley, CA 94704, USA
> http://www.icir.org/gregor/
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20110913/d1a30f7a/attachment.html