[Bro-Dev] #1038: Input framework memory usage issue with huge files

[Scott Campbell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Thanks - I will try to get at it tonight!

The workaround to the "reading huge files" problem seems like a good
one.  Now that you mention it, I did notice that before as well and
the description also explains a number of peculiar behaviors that I
have been seeing with the input framework.

cheers,
scott

On 7/15/13 4:00 PM, Bro Tracker wrote:
> #1038: Input framework memory usage issue with huge files 
> ------------------------+------------------------ Reporter:
> scampbell  |      Owner:  amannb Type:  Problem    |     Status:
> accepted Priority:  Medium     |  Milestone:  Bro2.2 Component:
> Bro        |    Version:  git/master Resolution:             |
> Keywords: ------------------------+------------------------ Changes
> (by amannb):
> 
> * priority:  High => Medium
> 
> 
> Comment:
> 
> Fastpath contains a fix for the memory leak -- could you check if
> that fixes your problem?
> 
> However, there is another issue with the input framework at the
> moment. While testing the memory leak, I noticed that Bro memory
> usage will spike while reading huge files. The reason for this is,
> that the input framework will add data to the
> child-thread->main-thread queue much faster than it is being read,
> which can mean that for a file of several hundred Megabytes,
> several GB of main memory are being held.
> 
> Robin and I think that solution for this is probably to make the
> child- thread->main-thread queue blocking once it reaches a certain
> size.
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.19 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlHkZbsACgkQK2Plq8B7ZBwnAwCgkBY5pbIeZSRLjH5+b9RnLkTD
8rAAn1SXnTYipsAsdMX870e5TAX8OqrJ
=J563
-----END PGP SIGNATURE-----