[Bro-Dev] #1038: Input framework memory usage issue with huge files

Bro Tracker bro at tracker.bro.org
Mon Jul 15 14:15:57 PDT 2013


#1038: Input framework memory usage issue with huge files
------------------------+------------------------
  Reporter:  scampbell  |      Owner:  amannb
      Type:  Problem    |     Status:  accepted
  Priority:  Medium     |  Milestone:  Bro2.2
 Component:  Bro        |    Version:  git/master
Resolution:             |   Keywords:
------------------------+------------------------

Comment (by scampbell):

 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1

 Thanks - I will try to get at it tonight!

 The workaround to the "reading huge files" problem seems like a good
 one.  Now that you mention it, I did notice that before as well and
 the description also explains a number of peculiar behaviors that I
 have been seeing with the input framework.

 cheers,
 scott

 On 7/15/13 4:00 PM, Bro Tracker wrote:
 > #1038: Input framework memory usage issue with huge files
 > ------------------------+------------------------ Reporter:
 > scampbell  |      Owner:  amannb Type:  Problem    |     Status:
 > accepted Priority:  Medium     |  Milestone:  Bro2.2 Component:
 > Bro        |    Version:  git/master Resolution:             |
 > Keywords: ------------------------+------------------------ Changes
 > (by amannb):
 >
 > * priority:  High => Medium
 >
 >
 > Comment:
 >
 > Fastpath contains a fix for the memory leak -- could you check if
 > that fixes your problem?
 >
 > However, there is another issue with the input framework at the
 > moment. While testing the memory leak, I noticed that Bro memory
 > usage will spike while reading huge files. The reason for this is,
 > that the input framework will add data to the
 > child-thread->main-thread queue much faster than it is being read,
 > which can mean that for a file of several hundred Megabytes,
 > several GB of main memory are being held.
 >
 > Robin and I think that solution for this is probably to make the
 > child- thread->main-thread queue blocking once it reaches a certain
 > size.
 >

 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG/MacGPG2 v2.0.19 (Darwin)
 Comment: GPGTools - http://gpgtools.org
 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

 iEYEARECAAYFAlHkZbsACgkQK2Plq8B7ZBwnAwCgkBY5pbIeZSRLjH5+b9RnLkTD
 8rAAn1SXnTYipsAsdMX870e5TAX8OqrJ
 =J563
 -----END PGP SIGNATURE-----

-- 
Ticket URL: <http://tracker.bro.org/bro/ticket/1038#comment:7>
Bro Tracker <http://tracker.bro.org/bro>
Bro Issue Tracker



More information about the bro-dev mailing list