[Bro-Dev] #1038: Input framework memory usage issue with huge files

[Scott Campbell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 7/15/13 4:00 PM, Bro Tracker wrote:
> #1038: Input framework memory usage issue with huge files 
> ------------------------+------------------------ Reporter:
> scampbell  |      Owner:  amannb Type:  Problem    |     Status:
> accepted Priority:  Medium     |  Milestone:  Bro2.2 Component:
> Bro        |    Version:  git/master Resolution:             |
> Keywords: ------------------------+------------------------ Changes
> (by amannb):
> 
> * priority:  High => Medium
> 
> 
> Comment:
> 
> Fastpath contains a fix for the memory leak -- could you check if
> that fixes your problem?
> 
> However, there is another issue with the input framework at the
> moment. While testing the memory leak, I noticed that Bro memory
> usage will spike while reading huge files. The reason for this is,
> that the input framework will add data to the
> child-thread->main-thread queue much faster than it is being read,
> which can mean that for a file of several hundred Megabytes,
> several GB of main memory are being held.
> 
> Robin and I think that solution for this is probably to make the
> child- thread->main-thread queue blocking once it reaches a certain
> size.
> 

I am seeing some significant issues with this run (2.1-814).  When I
start with a new install (no session state) with an empty but existing
data file the exec size rapidly goes to 9-16G, the CPU runs at 100%
and any additional data added to the data file after start is ignored.

The test is being done on OSX 10.8.4 .

Sorry for the bad news,
scott
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.19 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlHlkKcACgkQK2Plq8B7ZBy91gCgtxlreSjH10IqzFz50f5Fs1L8
qLgAoLR36AfLizWI8VZdLr4pr4xLxyCO
=GXur
-----END PGP SIGNATURE-----