[Bro-Dev] [JIRA] (BIT-1182) Input-framework thread spwan

[Johanna Amann (JIRA)]

    [ https://bro-tracker.atlassian.net/browse/BIT-1182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=19907#comment-19907 ] 

Johanna Amann commented on BIT-1182:
------------------------------------

I am actually not sure if this is an input framework issue. The input framework does not spawn a new thread for each change in an input file. Instead, all the changes are serialized into events by a single input reader, which is responsible for the file. For 5,000 changed lines, this should be rather fast - it probably processes all changes in less then a second.

If I understood everything correctly, things work as long as you do not use the exec framework. The problem here is actually that the exec framework spawns a thread for each execution that you want to perform (...because one input reader is spawned per execution...). As you get all change events near-simultaneously, all of them are spawned near-simultaneously - and I can see that leading to all kinds of problems.

I am not quite sure what the best way to handle that is though. Throttling the number of events that the input framework sends should be possible -- however, I am not sure if it is desirable since it should usually work without too much troubles. You would run into the same problem if you try to spawn a lot of exec framework tasks because of some other event (e.g. a lot of network packets that trigger it at the same time).

> Input-framework thread spwan
> ----------------------------
>
>                 Key: BIT-1182
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1182
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: 2.2
>            Reporter: Aashish Sharma
>              Labels: input-framework
>
> Using the mode REREAD, I noticed that input-framework spawns a thread for every add/change/delete for the elements in the feed file. 
> this is a VERY desired feature and powerful capability and works quite well in general settings. 
> Since, all the changes in a file spawns a thread to process for: EVENT_NEW, EVENT_CHANGED, EVENT_REMOVED, If there are lets say 5000 Changes in the file, there would be 5000 threads spawned at the same time. this is still alright and system can handle load and processing is done in a few seconds.
> However, if I include a when statement along with exec framework usage to execute an action in Input::EVENT_NEW, Input::EVENT_CHANGED or Input::EVENT_REMOVED - all threads spawned together freezes bro from processing any packets at all. 
> It would be nice if we can serialize this thread creation and spawn only a few at a time. This way we can spread out the increased load over next N mins instead of freezing bro to a standstill. 
> (As always, please let me know if you want code to be able to re-produce this issue). 



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)