[Bro] log streams in a bro cluster

[Azoff, Justin S]

> On Jun 9, 2016, at 7:16 AM, Luis Martin Liras <martin.liras at gmail.com> wrote:
> 
> Hi all,
> 
> A little bit of investigation and I found (with a tcpdump) that the logs arrive to the manager process BUT they are not stored to disk. Then I found the following entry at the beginning of the communication.log file :
> 
> 1465472892.006482    manager    parent    -    -    -    error [#10002/192.168.1.10:57322] unserializing event/function Notice::cluster_notice: write error for creating writer
> 
> 
> followed by a lot of errors like:
> 
> 1465473767.549373    manager    parent    -    -    -    error [#10001/192.168.1.10:57322] unserializing event/function Notice::cluster_notice: write error for log entry
> 

Ah.. the "write error for creating writer" message is a bit misleading, it outputs that for any error in the process.

Those messages also point to an issue with notices, not with your log file.  Are you also calling NOTICE somewhere?

Your problem could be that there is a discrepancy between how you defined warn_info and what you are passing Log::write.  Non clustered bro doesn't need to serialize/deserialize the messages so you can get away with certain mistakes that break once you use a cluster.

The standard log files all use the same mechanism, so if you are getting an http.log then your remote communication is working and there should be nothing preventing your log file from being written.

It would help if you could post your scripts somewhere or try to come up with a minimal example that shows the problem.

This is the simplest example for writing a custom log file:

http://try.bro.org/#/trybro?example=log

If you modify it like this and deploy it to a cluster you should get a foo.log containing things like

1465477100.871640	hello from manager
1465477105.884494	hello from manager
1465477104.537564	hello from proxy-1
1465477108.648193	hello from worker-1-2
1465477108.527117	hello from worker-1-1
1465477110.887240	hello from manager
1465477113.652352	hello from worker-1-2
1465477109.552765	hello from proxy-1



module FOO;

export {
    redef enum Log::ID += { LOG };

    type Info: record {
        ts: time &log;
        msg: string &log;
    };
}

event do_log()
    {
    local l = [$ts = network_time(), $msg=fmt("hello from %s", peer_description)];
    Log::write(LOG, l);
    schedule 5sec {do_log() };
    }

event bro_init()
{
    Log::create_stream(LOG, [$columns=Info]);

    schedule 5sec {do_log() };
}



-- 
- Justin Azoff