[Bro-Dev] #1038: Input framework memory usage issue with huge files (was: memory leak in input framework)

[Bro Tracker]

#1038: Input framework memory usage issue with huge files
------------------------+------------------------
  Reporter:  scampbell  |      Owner:  amannb
      Type:  Problem    |     Status:  accepted
  Priority:  Medium     |  Milestone:  Bro2.2
 Component:  Bro        |    Version:  git/master
Resolution:             |   Keywords:
------------------------+------------------------
Changes (by amannb):

 * priority:  High => Medium


Comment:

 Fastpath contains a fix for the memory leak -- could you check if that
 fixes your problem?

 However, there is another issue with the input framework at the moment.
 While testing the memory leak, I noticed that Bro memory usage will spike
 while reading huge files. The reason for this is, that the input framework
 will add data to the child-thread->main-thread queue much faster than it
 is being read, which can mean that for a file of several hundred
 Megabytes, several GB of main memory are being held.

 Robin and I think that solution for this is probably to make the child-
 thread->main-thread queue blocking once it reaches a certain size.

-- 
Ticket URL: <http://tracker.bro.org/bro/ticket/1038#comment:3>
Bro Tracker <http://tracker.bro.org/bro>
Bro Issue Tracker