[Zeek] zeekctl netstats returns time out
Carlos Lopez
clopmz at outlook.com
Wed Feb 19 00:24:06 PST 2020
Good morning,
Many thanks for your help Jon. All my config that you have requested.
- node.cfg:
[manager]
type=manager
host=localhost
[logger]
type=logger
host=localhost
[proxy-1]
type=proxy
host=localhost
[worker-1]
type=worker
host=localhost
interface=netmap:vtnet2
[worker-2]
type=worker
host=localhost
interface=netmap:vtnet3
- sockstat -l4:
USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS
root sendmail 50520 3 tcp4 127.0.0.1:25 *:*
root sshd 31667 3 tcp4 *:22 *:*
root zeek 27934 17 tcp46 *:47765 *:*
root zeek 20657 17 tcp46 *:47764 *:*
root zeek 84818 16 tcp46 *:47763 *:*
root zeek 91782 16 tcp46 *:47762 *:*
root zeek 94252 17 tcp46 *:47761 *:*
root owlhnode 334 6 tcp46 *:50002 *:*
root nfsd 46617 5 tcp4 *:2049 *:*
root mountd 37746 8 udp4 *:650 *:*
root mountd 37746 9 tcp4 *:650 *:*
root rpcbind 52182 9 udp4 *:111 *:*
root rpcbind 52182 10 udp4 *:947 *:*
root rpcbind 52182 11 tcp4 *:111 *:*
? ? ? ? udp4 *:2049 *:*
- nc command (also ipv6 works):
root at fbsdzeek01:~ # nc -zv 127.0.0.1 47761
Connection to 127.0.0.1 47761 port [tcp/*] succeeded!
- broker.log:
{"ts":"2020-02-19T08:13:35.215215Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10007,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:35.214435Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"127.0.0.1","peer.bound_port":47761,"message":"received handshake from remote core"}
{"ts":"2020-02-19T08:13:37.198510Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10008,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:37.198165Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10009,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:36.965614Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"127.0.0.1","peer.bound_port":47761,"message":"received handshake from remote core"}
{"ts":"2020-02-19T08:13:41.269695Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10010,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:41.275816Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10011,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:36.965614Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"127.0.0.1","peer.bound_port":47762,"message":"received handshake from remote core"}
{"ts":"2020-02-19T08:13:41.271616Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10012,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:41.505503Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10013,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:41.579196Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"127.0.0.1","peer.bound_port":47761,"message":"received handshake from remote core"}
{"ts":"2020-02-19T08:13:41.964270Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"127.0.0.1","peer.bound_port":47761,"message":"received handshake from remote core"}
As you can see, nothing strange here ... As you said, I have changed the definition of "localhost" in the node.cfg file to IP 127.0.0.1 ... and it works!
Problem solved. Many thanks Jon...
Regards,
C. L. Martinez
________________________________________
From: Jon Siwek <jsiwek at corelight.com>
Sent: 18 February 2020 20:16
To: Carlos Lopez
Cc: zeek at zeek.org
Subject: Re: [Zeek] zeekctl netstats returns time out
Since this is working in my own environment, we could maybe compare
configs until we find the differences. What's the node.cfg you use?
If it's all just a single node using localhost, these are some of the
first things that come to mind for troubleshooting:
Confirm TCP connectivity:
# nc -zv 127.0.0.1 47761
Connection to localhost 47761 port [tcp/*] succeeded!
There's also the other 47761+ ports to try, but likely all get the
same result as the first one. An IPv4 vs. IPv6 config issue might
also be a problem and can try variations of "::1" and "localhost" in
place of "127.0.0.1" if it's all one node. To really get all IPv4,
think you can even set 127.0.0.1 in node.cfg and run like this:
ZEEK_DEFAULT_LISTEN_ADDRESS=127.0.0.1 /usr/local/zeek/bin/zeekctl deploy
The high-level connection attempts are also logged here:
/usr/local/zeek/logs/current/broker.log
See anything interesting there? It should have several initial
"peer-added" and "handshake successful" entries for the initial
cluster setup and then for each time you try something like `zeekctl
netstats worker-1` it will have a pair of "peer-added" and
"connection-terminated" entries.
- Jon
On Tue, Feb 18, 2020 at 5:53 AM Carlos Lopez <clopmz at outlook.com> wrote:
>
> Any idea about how to debug this error?
>
>
> Regards,
> C. L. Martinez
>
>
> ________________________________________
> From: zeek-bounces at zeek.org <zeek-bounces at zeek.org> on behalf of Carlos Lopez <clopmz at outlook.com>
> Sent: 15 February 2020 23:21
> To: Jon Siwek
> Cc: zeek at zeek.org
> Subject: Re: [Zeek] zeekctl netstats returns time out
>
> Many thanks Jon. Regarding TCP connectivity, I have neither ipfw nor pf enabled between manager and workers. And respecting to "busy" system, shouldn't be the problem either. For example, my top output in standalone config:
>
> last pid: 6492; load averages: 0.16, 0.22, 0.22 up 0+06:21:48 22:20:43
> 44 threads: 1 running, 43 sleeping
> CPU: 0.0% user, 0.0% nice, 1.9% system, 0.0% interrupt, 98.1% idle
> Mem: 51M Active, 58M Inact, 679M Wired, 271M Buf, 5137M Free
> Swap: 4096M Total, 4096M Free
>
> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
> 45091 root 22 0 460M 111M select 1 18:29 4.71% zeek{zeek}
> 6492 root 20 0 1044M 4144K CPU0 0 0:00 0.05% top
> 45091 root 20 0 460M 111M uwait 0 0:22 0.02% zeek{caf.clock}
> 39952 _ntp 20 -20 1038M 4000K select 1 0:03 0.01% ntpd
> 45407 root 20 0 1044M 9912K select 1 0:00 0.01% sshd
> 45091 root 20 0 460M 111M uwait 1 0:09 0.01% zeek{caf.multiplexer}
> 45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.ntp/Log::WRITER_}
> 45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.files/Log::WRITE}
> 45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.capture_loss/Log}
> 45091 root 20 0 460M 111M uwait 1 0:01 0.00% zeek{zk.dns/Log::WRITER_}
> 45091 root 20 0 460M 111M uwait 1 0:01 0.00% zeek{zk.ssl/Log::WRITER_}
> 45091 root 20 0 460M 111M uwait 1 0:01 0.00% zeek{zk.http/Log::WRITER}
> 45091 root 20 0 460M 111M uwait 1 0:01 0.00% zeek{zk.loaded_scripts/L}
> 45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.packet_filter/Lo}
> 45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.stats/Log::WRITE}
> 45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.conn/Log::WRITER}
> 45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.software/Log::WR}
> 45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.known_services/L}
> 45091 root 20 0 460M 111M uwait 1 0:01 0.00% zeek{zk.x509/Log::WRITER}
> 45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.notice/Log::WRIT}
> 45091 root 20 0 460M 111M uwait 1 0:01 0.00% zeek{zk.ssh/Log::WRITER_}
> 45091 root 20 0 460M 111M uwait 1 0:01 0.00% zeek{zk.kerberos/Log::WR}
> 45091 root 20 0 460M 111M uwait 1 0:01 0.00% zeek{zk.broker/Log::WRIT}
> 45091 root 20 0 460M 111M uwait 1 0:01 0.00% zeek{zk.weird/Log::WRITE}
> 45091 root 20 0 460M 111M uwait 0 0:00 0.00% zeek{zk.dhcp/Log::WRITER}
> 45091 root 20 0 460M 111M uwait 1 0:00 0.00% zeek{zk.known_certs/Log:}
> 45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.known_hosts/Log:}
> 96485 root 20 0 17M 6920K select 0 0:00 0.00% sendmail
> 45091 root 20 0 460M 111M select 0 0:00 0.00% zeek{caf.multiplexer}
>
>
> --
> Regards,
> C. L. Martinez
>
> On 15/02/2020, 18:57, "Jon Siwek" <jsiwek at corelight.com> wrote:
>
> Zeek 3.0.1's `zeekctl netstats` is working for me in FreeBSD 12.1.
> TCP connectivity is required for that command to work and you can read
> more about the ports involved for further troubleshooting here:
>
> https://github.com/zeek/zeekctl#zeek-communication
>
> If the Zeek processes are particularly busy, that could also be a
> reason for timing out. The `CommTimeout` (default 10 seconds) can be
> increased in `zeekctl.cfg` in that case.
>
> - Jon
>
> On Sat, Feb 15, 2020 at 8:46 AM Carlos Lopez <clopmz at outlook.com> wrote:
> >
> > Hi all,
> >
> >
> >
> > Every time I run “zeekctl netstats” returns time out under FreeBSD 12.1 hosts using netmap:
> >
> >
> >
> > root at fbsdzeek01:/nsm/zeek/logs/current # zeekctl netstats
> >
> >
> >
> > Warning: ZeekControl plugin uses legacy BroControl API. Use
> >
> > 'import ZeekControl.plugin' instead of 'import BroControl.plugin'
> >
> >
> >
> > zeek: <error: time-out>
> >
> >
> >
> > This behavior occurs in both standalone and cluster configurations. Any idea? Maybe is it a bug?
> >
> >
> >
> > --
> >
> > Regards,
> >
> > C. L. Martinez
> >
> > _______________________________________________
> > Zeek mailing list
> > zeek at zeek.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
More information about the Zeek
mailing list