[Bro] HTTP Object length calculation
gregor at icir.org
Tue Sep 13 13:12:50 PDT 2011
On 9/13/11 10:05 , Seth Hall wrote:
> On Sep 13, 2011, at 11:57 AM, Ioannis WiCom wrote:
>> I have isolated an example TCP connection, and measured the bytes using wireshark. The real object length is equal to the "Content-Length", but the reported by bro is much higher. Therefore, I cannot understand what the value stat$body_length represents.
> stat$body_length *should* be the actual counted number of bytes that were in the body. If you see a disparity between the two numbers, the web server could be reporting an incorrect length for the data it's sending. Could you send the trace file privately?
Actually that's not exactly the case. Bro reports the body length *after
decompression* (for transfer-encodings that use compressions).
In addition, the Content-Length header is often unreliable. E.g., if an
HTTP transfer is interrupted fewer bytes are transferred that reported
by Content-Length. This can happen often with (misconfigured) download
managers. Or the Content-Length header can also be just plain wrong
(HTTP server sends garbage). We did a study with residential traffic and
found that the Content-Length header will on average over-report the
volume by a factor of about 5 (with some spikes reaching several 100(!))
<gregor at icir.org> <gregor at icsi.berkeley.edu>
Int. Computer Science Institute (ICSI)
1947 Center St., Ste. 600
Berkeley, CA 94704, USA
More information about the Bro