[Zeek] zeekctl netstats returns time out
Jon Siwek
jsiwek at corelight.com
Tue Feb 18 11:16:57 PST 2020
Since this is working in my own environment, we could maybe compare
configs until we find the differences. What's the node.cfg you use?
If it's all just a single node using localhost, these are some of the
first things that come to mind for troubleshooting:
Confirm TCP connectivity:
# nc -zv 127.0.0.1 47761
Connection to localhost 47761 port [tcp/*] succeeded!
There's also the other 47761+ ports to try, but likely all get the
same result as the first one. An IPv4 vs. IPv6 config issue might
also be a problem and can try variations of "::1" and "localhost" in
place of "127.0.0.1" if it's all one node. To really get all IPv4,
think you can even set 127.0.0.1 in node.cfg and run like this:
ZEEK_DEFAULT_LISTEN_ADDRESS=127.0.0.1 /usr/local/zeek/bin/zeekctl deploy
The high-level connection attempts are also logged here:
/usr/local/zeek/logs/current/broker.log
See anything interesting there? It should have several initial
"peer-added" and "handshake successful" entries for the initial
cluster setup and then for each time you try something like `zeekctl
netstats worker-1` it will have a pair of "peer-added" and
"connection-terminated" entries.
- Jon
On Tue, Feb 18, 2020 at 5:53 AM Carlos Lopez <clopmz at outlook.com> wrote:
>
> Any idea about how to debug this error?
>
>
> Regards,
> C. L. Martinez
>
>
> ________________________________________
> From: zeek-bounces at zeek.org <zeek-bounces at zeek.org> on behalf of Carlos Lopez <clopmz at outlook.com>
> Sent: 15 February 2020 23:21
> To: Jon Siwek
> Cc: zeek at zeek.org
> Subject: Re: [Zeek] zeekctl netstats returns time out
>
> Many thanks Jon. Regarding TCP connectivity, I have neither ipfw nor pf enabled between manager and workers. And respecting to "busy" system, shouldn't be the problem either. For example, my top output in standalone config:
>
> last pid: 6492; load averages: 0.16, 0.22, 0.22 up 0+06:21:48 22:20:43
> 44 threads: 1 running, 43 sleeping
> CPU: 0.0% user, 0.0% nice, 1.9% system, 0.0% interrupt, 98.1% idle
> Mem: 51M Active, 58M Inact, 679M Wired, 271M Buf, 5137M Free
> Swap: 4096M Total, 4096M Free
>
> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
> 45091 root 22 0 460M 111M select 1 18:29 4.71% zeek{zeek}
> 6492 root 20 0 1044M 4144K CPU0 0 0:00 0.05% top
> 45091 root 20 0 460M 111M uwait 0 0:22 0.02% zeek{caf.clock}
> 39952 _ntp 20 -20 1038M 4000K select 1 0:03 0.01% ntpd
> 45407 root 20 0 1044M 9912K select 1 0:00 0.01% sshd
> 45091 root 20 0 460M 111M uwait 1 0:09 0.01% zeek{caf.multiplexer}
> 45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.ntp/Log::WRITER_}
> 45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.files/Log::WRITE}
> 45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.capture_loss/Log}
> 45091 root 20 0 460M 111M uwait 1 0:01 0.00% zeek{zk.dns/Log::WRITER_}
> 45091 root 20 0 460M 111M uwait 1 0:01 0.00% zeek{zk.ssl/Log::WRITER_}
> 45091 root 20 0 460M 111M uwait 1 0:01 0.00% zeek{zk.http/Log::WRITER}
> 45091 root 20 0 460M 111M uwait 1 0:01 0.00% zeek{zk.loaded_scripts/L}
> 45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.packet_filter/Lo}
> 45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.stats/Log::WRITE}
> 45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.conn/Log::WRITER}
> 45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.software/Log::WR}
> 45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.known_services/L}
> 45091 root 20 0 460M 111M uwait 1 0:01 0.00% zeek{zk.x509/Log::WRITER}
> 45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.notice/Log::WRIT}
> 45091 root 20 0 460M 111M uwait 1 0:01 0.00% zeek{zk.ssh/Log::WRITER_}
> 45091 root 20 0 460M 111M uwait 1 0:01 0.00% zeek{zk.kerberos/Log::WR}
> 45091 root 20 0 460M 111M uwait 1 0:01 0.00% zeek{zk.broker/Log::WRIT}
> 45091 root 20 0 460M 111M uwait 1 0:01 0.00% zeek{zk.weird/Log::WRITE}
> 45091 root 20 0 460M 111M uwait 0 0:00 0.00% zeek{zk.dhcp/Log::WRITER}
> 45091 root 20 0 460M 111M uwait 1 0:00 0.00% zeek{zk.known_certs/Log:}
> 45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.known_hosts/Log:}
> 96485 root 20 0 17M 6920K select 0 0:00 0.00% sendmail
> 45091 root 20 0 460M 111M select 0 0:00 0.00% zeek{caf.multiplexer}
>
>
> --
> Regards,
> C. L. Martinez
>
> On 15/02/2020, 18:57, "Jon Siwek" <jsiwek at corelight.com> wrote:
>
> Zeek 3.0.1's `zeekctl netstats` is working for me in FreeBSD 12.1.
> TCP connectivity is required for that command to work and you can read
> more about the ports involved for further troubleshooting here:
>
> https://github.com/zeek/zeekctl#zeek-communication
>
> If the Zeek processes are particularly busy, that could also be a
> reason for timing out. The `CommTimeout` (default 10 seconds) can be
> increased in `zeekctl.cfg` in that case.
>
> - Jon
>
> On Sat, Feb 15, 2020 at 8:46 AM Carlos Lopez <clopmz at outlook.com> wrote:
> >
> > Hi all,
> >
> >
> >
> > Every time I run “zeekctl netstats” returns time out under FreeBSD 12.1 hosts using netmap:
> >
> >
> >
> > root at fbsdzeek01:/nsm/zeek/logs/current # zeekctl netstats
> >
> >
> >
> > Warning: ZeekControl plugin uses legacy BroControl API. Use
> >
> > 'import ZeekControl.plugin' instead of 'import BroControl.plugin'
> >
> >
> >
> > zeek: <error: time-out>
> >
> >
> >
> > This behavior occurs in both standalone and cluster configurations. Any idea? Maybe is it a bug?
> >
> >
> >
> > --
> >
> > Regards,
> >
> > C. L. Martinez
> >
> > _______________________________________________
> > Zeek mailing list
> > zeek at zeek.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
More information about the Zeek
mailing list