[Bro-Dev] 0MQ security considerations

[Gilbert Clark]

Matthias Vallentin wrote:
>> That said, keep in mind that some libraries (e.g. DataSeries) actually 
>> spawn additional worker threads.  
>>     
>
> Is it possible to configure the number of threads DataSeries uses?
>   
Yes, to some extent; DataSeries will always spawn at least two threads 
(one output thread and one compression / worker thread). 

The number of worker threads DataSeries spawns can be controlled by 
tweaking a variable in dataseries.bro, but this brings up a more general 
problem: what if we use a library that doesn't offer us the ability to 
do this?

Then again, if the library doesn't let us do this, maybe we just 
shouldn't be using that particular library...?

>> I don't understand this bit.  Say we have 10 things to log: {A, B, C, 
>> ... J}, and we queue 8 messages before we flush + rotate (for the sake 
>> of argument)
>>
>> Log+Flush+Rotate task = {A, B, C, D}      ---> Thread 1
>> Log+Flush task = {E, F, G, H}   ---> Thread 2
>> Log+Flush+Finish task = {I, J}   ---> Thread 3
>>
>> In this case, it seems possible that thread 3 could complete before 
>> threads 1 and 2, unless we forced tasks to lock the log file upon 
>> execution. . . but if we did that, I think the locking order would 
>> become less predictable as the system's load increased, leading to oddly 
>> placed chunks of log data and / or dropped messages if the file were to 
>> be closed before a previous rotate message made it to the logger.
>>     
>
> This is a good example of hot to not use task-based parallelism. As
> there exist inter-task dependencies, they have to be processed
> sequentially. For example, say these 10 things to log are generated for
> 18 http_reply events. What I was saying is that you have one task for
> all dependent actions/messages/steps:
>
>     Event  1: Log+Flush+Rotate+Log+Flush+Finish task = {A, ..., J}
>     ...
>     Event 18: Log+Flush+Rotate+Log+Flush+Finish task = {A, ..., J}
>
> Say the Log Manager component has 3 threads associated, then each thread
> would process 6 events, assuming a simple uniform scheduling strategy.
>
>   

Sure.  In this case, though, A-J essentially compose the entire log 
file, which is what's giving me some trouble; even if the LogManager has 
3 threads associated, it still has to pass entire LogWriters around 
between those threads and make sure no two threads are using the same 
LogWriter (assuming nothing was using e.g. thread-local storage, which 
might stipulate that thread assignments would need to be made at 
initialization and, once made, could never change).

So, I suppose it might be good to spawn a finite number of Bro worker 
threads and assign LogWriters to those (rather than spawning one bro 
worker thread per logwriter), but I don't see a way around binding an 
entire logwriter to a thread.  What I don't like about this is that it 
means any scheduling the task manager does would be coarse-grained at best.

>> Kind of.  Let me throw together a diagram and send it out; it's 
>> something that should probably end up in the documentation anyway :)
>>     
>
> Sounds good. This stuff is much better to comprehend with a visual aid!
>   
Rough cut of a general overview is attached.  Probably going to change 
quite a bit, but I think it might help illustrate the way the system 
works overall.

The architecture, at the moment, is geared toward supporting transparent 
replacement of threaded log writers with writers that work across 
sockets and / or IPC; anything the implements QueueInterface can be used 
to pass messages back and forth.  Before this would actually work in a 
network / IPC context, though, the messages would need to support 
serialization; haven't done that yet because it hasn't really been a 
priority.

Feedback / corrections / "THAT'S A STUPID MODEL!" are always appreciated.

--Gilbert

-------------- next part --------------
A non-text attachment was scrubbed...
Name: LogSystemOverview.pdf.bz2
Type: application/x-bzip
Size: 897244 bytes
Desc: not available
Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20110728/4403b3d7/attachment-0001.bin