[Bro] log streams in a bro cluster

[Luis Martin Liras]

Hi all,

A little bit of investigation and I found (with a tcpdump) that the logs 
arrive to the manager process BUT they are not stored to disk. Then I 
found the following entry at the beginning of the communication.log file :

1465472892.006482    manager    parent    -    -    -    error 
[#10002/192.168.1.10:57322] unserializing event/function 
Notice::cluster_notice: write error for creating writer


followed by a lot of errors like:

1465473767.549373    manager    parent    -    -    -    error 
[#10001/192.168.1.10:57322] unserializing event/function 
Notice::cluster_notice: write error for log entry




It seems that for any reason, the deserializer writer is not able to 
open a writer ad then it's not possible to write the log files.

I found this error in github in RemoteSerializer.cc, in its function 
'ProcessLogCreateWriter':

https://github.com/bro/bro/blob/f5ce4785ea96b56643c092331a16308f071c8092/src/RemoteSerializer.cc



But I still cannot figure out why is this happening.


Tried to change permissions to all log directories to 777, but didn't work.

Any idea why could be happening this error?


Thank you!!


On 08/06/16 14:36, Azoff, Justin S wrote:
>> On Jun 8, 2016, at 4:43 AM, Luis Martin Liras <martin.liras at gmail.com> wrote:
>>
>> Hi all,
>>
>> I need some help with the logs generated by a Bro Cluster:
>>
>>
>> I have 5 bro scripts that run in all workers of my cluster
>> infrastructure. All of them work OK, sending notices to the manager and
>> all the staff, but one of them should create a LOG stream (warnings.log)
>> that I can't find anywhere:
>>
>> Log::create_stream(umas::WARN, [$columns=warn_info,$path="warnings"]);
>>
>> If I run my script in a single bro installation, all logs and notices
>> seem to work, but I need it working in a cluster infrastructure.
>>
>>
>> I expected this Log stream to be sent to the 'logs' directory in the
>> manager, but that log file is not there. Only standard log files
>> (dns.log, http.log, stdout.log, etc) are copied to the 'logs' directory.
>>
>> This warnings.log file do not appear either anywhere in the worker, and
>> not error log file is shown, so... I'm lost.
>>
>> I anyone can shed some light into this, I would appreciate it.
>>
> When are you writing to that log?  Just creating the log stream doesn't create the file until you do a
>
>      Log::write(umas::WARN, record);
>
>
>> The other problem I have is the following: My script should open a
>> config file. In a single machine infrastructure this config file is in
>> the same directory of the scripts, and everything work fine. The file is
>> opened and read. However in a cluster infrastructure the file is not
>> opened in the workers. I find that the file is copied by broctl to the
>> worker BUT it is not read when the bro script is running. Anyone can
>> tell me what I'm doing wrong or where should I locate that file in the
>> workers?
>>
>> Thank you for any help!!
> How are you loading the configuration file?
>
> You should be using something like
>
>      local config_path = fmt("%s/my-config.something", @DIR);
>
> otherwise a relative or absolute path may not be what you expect.
>