[Bro-Dev] 0MQ security considerations

[Matthias Vallentin]

> That said, keep in mind that some libraries (e.g. DataSeries) actually 
> spawn additional worker threads.  

Is it possible to configure the number of threads DataSeries uses?

> I don't understand this bit.  Say we have 10 things to log: {A, B, C, 
> ... J}, and we queue 8 messages before we flush + rotate (for the sake 
> of argument)
> 
> Log+Flush+Rotate task = {A, B, C, D}      ---> Thread 1
> Log+Flush task = {E, F, G, H}   ---> Thread 2
> Log+Flush+Finish task = {I, J}   ---> Thread 3
> 
> In this case, it seems possible that thread 3 could complete before 
> threads 1 and 2, unless we forced tasks to lock the log file upon 
> execution. . . but if we did that, I think the locking order would 
> become less predictable as the system's load increased, leading to oddly 
> placed chunks of log data and / or dropped messages if the file were to 
> be closed before a previous rotate message made it to the logger.

This is a good example of hot to not use task-based parallelism. As
there exist inter-task dependencies, they have to be processed
sequentially. For example, say these 10 things to log are generated for
18 http_reply events. What I was saying is that you have one task for
all dependent actions/messages/steps:

    Event  1: Log+Flush+Rotate+Log+Flush+Finish task = {A, ..., J}
    ...
    Event 18: Log+Flush+Rotate+Log+Flush+Finish task = {A, ..., J}

Say the Log Manager component has 3 threads associated, then each thread
would process 6 events, assuming a simple uniform scheduling strategy.

> Kind of.  Let me throw together a diagram and send it out; it's 
> something that should probably end up in the documentation anyway :)

Sounds good. This stuff is much better to comprehend with a visual aid!

> I was more attempting to illustrate the parallel between bro's existing 
> cluster architecture and a more traditional task-oriented model (as I 
> understand it) than I was trying to argue that packets should 
> necessarily be classifiable as "tasks" in the strictest sense of the word.

Yup, I understood it that way and think it's a nice analogy.

    Matthias