[Bro] HTTP Object length calculation

[Gregor Maier]

On 9/13/11 10:05 , Seth Hall wrote:
>
> On Sep 13, 2011, at 11:57 AM, Ioannis WiCom wrote:
>
>> I have isolated an example TCP connection, and measured the bytes using wireshark. The real object length is equal to the "Content-Length", but the reported by bro is much higher. Therefore, I cannot understand what the value stat$body_length represents.
>
> stat$body_length *should* be the actual counted number of bytes that were in the body.  If you see a disparity between the two numbers, the web server could be reporting an incorrect length for the data it's sending.  Could you send the trace file privately?

Actually that's not exactly the case. Bro reports the body length *after 
decompression* (for transfer-encodings that use compressions).

In addition, the Content-Length header is often unreliable. E.g., if an 
HTTP transfer is interrupted fewer bytes are transferred that reported 
by Content-Length. This can happen often with (misconfigured) download 
managers. Or the Content-Length header can also be just plain wrong 
(HTTP server sends garbage). We did a study with residential traffic and 
found that the Content-Length header will on average over-report the 
volume by a factor of about 5 (with some spikes reaching several 100(!))

cu
Gregor
-- 
Gregor Maier
<gregor at icir.org>  <gregor at icsi.berkeley.edu>
Int. Computer Science Institute (ICSI)
1947 Center St., Ste. 600
Berkeley, CA 94704, USA
http://www.icir.org/gregor/