[Bro-Dev] #1038: Input framework memory usage issue with huge files

[Bro Tracker]

#1038: Input framework memory usage issue with huge files
------------------------+------------------------
  Reporter:  scampbell  |      Owner:  amannb
      Type:  Problem    |     Status:  accepted
  Priority:  Medium     |  Milestone:  Bro2.2
 Component:  Bro        |    Version:  git/master
Resolution:             |   Keywords:
------------------------+------------------------

Comment (by scampbell):

 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1

 On 7/15/13 4:00 PM, Bro Tracker wrote:
 > #1038: Input framework memory usage issue with huge files
 > ------------------------+------------------------ Reporter:
 > scampbell  |      Owner:  amannb Type:  Problem    |     Status:
 > accepted Priority:  Medium     |  Milestone:  Bro2.2 Component:
 > Bro        |    Version:  git/master Resolution:             |
 > Keywords: ------------------------+------------------------ Changes
 > (by amannb):
 >
 > * priority:  High => Medium
 >
 >
 > Comment:
 >
 > Fastpath contains a fix for the memory leak -- could you check if
 > that fixes your problem?
 >
 > However, there is another issue with the input framework at the
 > moment. While testing the memory leak, I noticed that Bro memory
 > usage will spike while reading huge files. The reason for this is,
 > that the input framework will add data to the
 > child-thread->main-thread queue much faster than it is being read,
 > which can mean that for a file of several hundred Megabytes,
 > several GB of main memory are being held.
 >
 > Robin and I think that solution for this is probably to make the
 > child- thread->main-thread queue blocking once it reaches a certain
 > size.
 >

 I am seeing some significant issues with this run (2.1-814).  When I
 start with a new install (no session state) with an empty but existing
 data file the exec size rapidly goes to 9-16G, the CPU runs at 100%
 and any additional data added to the data file after start is ignored.

 The test is being done on OSX 10.8.4 .

 Sorry for the bad news,
 scott
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG/MacGPG2 v2.0.19 (Darwin)
 Comment: GPGTools - http://gpgtools.org
 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

 iEYEARECAAYFAlHlkKcACgkQK2Plq8B7ZBy91gCgtxlreSjH10IqzFz50f5Fs1L8
 qLgAoLR36AfLizWI8VZdLr4pr4xLxyCO
 =GXur
 -----END PGP SIGNATURE-----

-- 
Ticket URL: <http://tracker.bro.org/bro/ticket/1038#comment:8>
Bro Tracker <http://tracker.bro.org/bro>
Bro Issue Tracker