[Zeek] zeekctl netstats returns time out

Carlos Lopez clopmz at outlook.com
Wed Feb 19 00:24:06 PST 2020


Good morning,

 Many thanks for your help Jon. All my config that you have requested.

- node.cfg:

[manager]
type=manager
host=localhost

[logger]
type=logger
host=localhost

[proxy-1]
type=proxy
host=localhost

[worker-1]
type=worker
host=localhost
interface=netmap:vtnet2

[worker-2]
type=worker
host=localhost
interface=netmap:vtnet3

- sockstat -l4:
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
root     sendmail   50520 3  tcp4   127.0.0.1:25          *:*
root     sshd       31667 3  tcp4   *:22                  *:*
root     zeek       27934 17 tcp46  *:47765               *:*
root     zeek       20657 17 tcp46  *:47764               *:*
root     zeek       84818 16 tcp46  *:47763               *:*
root     zeek       91782 16 tcp46  *:47762               *:*
root     zeek       94252 17 tcp46  *:47761               *:*
root     owlhnode   334   6  tcp46  *:50002               *:*
root     nfsd       46617 5  tcp4   *:2049                *:*
root     mountd     37746 8  udp4   *:650                 *:*
root     mountd     37746 9  tcp4   *:650                 *:*
root     rpcbind    52182 9  udp4   *:111                 *:*
root     rpcbind    52182 10 udp4   *:947                 *:*
root     rpcbind    52182 11 tcp4   *:111                 *:*
?        ?          ?     ?  udp4   *:2049                *:*

- nc command (also ipv6 works):

root at fbsdzeek01:~ # nc -zv 127.0.0.1 47761
Connection to 127.0.0.1 47761 port [tcp/*] succeeded!

 - broker.log:

{"ts":"2020-02-19T08:13:35.215215Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10007,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:35.214435Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"127.0.0.1","peer.bound_port":47761,"message":"received handshake from remote core"}
{"ts":"2020-02-19T08:13:37.198510Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10008,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:37.198165Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10009,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:36.965614Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"127.0.0.1","peer.bound_port":47761,"message":"received handshake from remote core"}
{"ts":"2020-02-19T08:13:41.269695Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10010,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:41.275816Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10011,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:36.965614Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"127.0.0.1","peer.bound_port":47762,"message":"received handshake from remote core"}
{"ts":"2020-02-19T08:13:41.271616Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10012,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:41.505503Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10013,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:41.579196Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"127.0.0.1","peer.bound_port":47761,"message":"received handshake from remote core"}
{"ts":"2020-02-19T08:13:41.964270Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"127.0.0.1","peer.bound_port":47761,"message":"received handshake from remote core"}

 As you can see, nothing strange here ... As you said, I have changed the definition of "localhost" in the node.cfg file to IP 127.0.0.1 ... and it works!

 Problem solved. Many thanks Jon...

Regards,
C. L. Martinez


________________________________________
From: Jon Siwek <jsiwek at corelight.com>
Sent: 18 February 2020 20:16
To: Carlos Lopez
Cc: zeek at zeek.org
Subject: Re: [Zeek] zeekctl netstats returns time out

Since this is working in my own environment, we could maybe compare
configs until we find the differences.  What's the node.cfg you use?
If it's all just a single node using localhost, these are some of the
first things that come to mind for troubleshooting:

Confirm TCP connectivity:

# nc -zv 127.0.0.1 47761
Connection to localhost 47761 port [tcp/*] succeeded!

There's also the other 47761+ ports to try, but likely all get the
same result as the first one.  An IPv4 vs. IPv6 config issue might
also be a problem and can try variations of "::1" and "localhost" in
place of "127.0.0.1" if it's all one node.  To really get all IPv4,
think you can even set 127.0.0.1 in node.cfg and run like this:

    ZEEK_DEFAULT_LISTEN_ADDRESS=127.0.0.1 /usr/local/zeek/bin/zeekctl deploy

The high-level connection attempts are also logged here:

    /usr/local/zeek/logs/current/broker.log

See anything interesting there?  It should have several initial
"peer-added" and "handshake successful" entries for the initial
cluster setup and then for each time you try something like `zeekctl
netstats worker-1` it will have a pair of "peer-added" and
"connection-terminated" entries.

- Jon

On Tue, Feb 18, 2020 at 5:53 AM Carlos Lopez <clopmz at outlook.com> wrote:
>
> Any idea about how to debug this error?
>
>
> Regards,
> C. L. Martinez
>
>
> ________________________________________
> From: zeek-bounces at zeek.org <zeek-bounces at zeek.org> on behalf of Carlos Lopez <clopmz at outlook.com>
> Sent: 15 February 2020 23:21
> To: Jon Siwek
> Cc: zeek at zeek.org
> Subject: Re: [Zeek] zeekctl netstats returns time out
>
> Many thanks Jon. Regarding TCP connectivity, I have neither ipfw nor pf enabled between manager and workers. And respecting to "busy" system, shouldn't be the problem either. For example, my top output in standalone config:
>
> last pid:  6492;  load averages:  0.16,  0.22,  0.22                                                                                                                                up 0+06:21:48  22:20:43
> 44 threads:    1 running, 43 sleeping
> CPU:  0.0% user,  0.0% nice,  1.9% system,  0.0% interrupt, 98.1% idle
> Mem: 51M Active, 58M Inact, 679M Wired, 271M Buf, 5137M Free
> Swap: 4096M Total, 4096M Free
>
>   PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
> 45091 root         22    0   460M   111M select   1  18:29   4.71% zeek{zeek}
>  6492 root         20    0  1044M  4144K CPU0     0   0:00   0.05% top
> 45091 root         20    0   460M   111M uwait    0   0:22   0.02% zeek{caf.clock}
> 39952 _ntp         20  -20  1038M  4000K select   1   0:03   0.01% ntpd
> 45407 root         20    0  1044M  9912K select   1   0:00   0.01% sshd
> 45091 root         20    0   460M   111M uwait    1   0:09   0.01% zeek{caf.multiplexer}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.ntp/Log::WRITER_}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.files/Log::WRITE}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.capture_loss/Log}
> 45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.dns/Log::WRITER_}
> 45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.ssl/Log::WRITER_}
> 45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.http/Log::WRITER}
> 45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.loaded_scripts/L}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.packet_filter/Lo}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.stats/Log::WRITE}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.conn/Log::WRITER}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.software/Log::WR}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.known_services/L}
> 45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.x509/Log::WRITER}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.notice/Log::WRIT}
> 45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.ssh/Log::WRITER_}
> 45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.kerberos/Log::WR}
> 45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.broker/Log::WRIT}
> 45091 root         20    0   460M   111M uwait    1   0:01   0.00% zeek{zk.weird/Log::WRITE}
> 45091 root         20    0   460M   111M uwait    0   0:00   0.00% zeek{zk.dhcp/Log::WRITER}
> 45091 root         20    0   460M   111M uwait    1   0:00   0.00% zeek{zk.known_certs/Log:}
> 45091 root         20    0   460M   111M uwait    0   0:01   0.00% zeek{zk.known_hosts/Log:}
> 96485 root         20    0    17M  6920K select   0   0:00   0.00% sendmail
> 45091 root         20    0   460M   111M select   0   0:00   0.00% zeek{caf.multiplexer}
>
>
> --
> Regards,
> C. L. Martinez
>
> On 15/02/2020, 18:57, "Jon Siwek" <jsiwek at corelight.com> wrote:
>
>     Zeek 3.0.1's `zeekctl netstats` is working for me in FreeBSD 12.1.
>     TCP connectivity is required for that command to work and you can read
>     more about the ports involved for further troubleshooting here:
>
>         https://github.com/zeek/zeekctl#zeek-communication
>
>     If the Zeek processes are particularly busy, that could also be a
>     reason for timing out.  The `CommTimeout` (default 10 seconds) can be
>     increased in `zeekctl.cfg` in that case.
>
>     - Jon
>
>     On Sat, Feb 15, 2020 at 8:46 AM Carlos Lopez <clopmz at outlook.com> wrote:
>     >
>     > Hi all,
>     >
>     >
>     >
>     > Every time I run “zeekctl netstats” returns time out under FreeBSD 12.1 hosts using netmap:
>     >
>     >
>     >
>     > root at fbsdzeek01:/nsm/zeek/logs/current # zeekctl netstats
>     >
>     >
>     >
>     > Warning: ZeekControl plugin uses legacy BroControl API. Use
>     >
>     > 'import ZeekControl.plugin' instead of 'import BroControl.plugin'
>     >
>     >
>     >
>     >        zeek: <error: time-out>
>     >
>     >
>     >
>     > This behavior occurs in both standalone and cluster configurations. Any idea? Maybe is it a bug?
>     >
>     >
>     >
>     > --
>     >
>     > Regards,
>     >
>     > C. L. Martinez
>     >
>     > _______________________________________________
>     > Zeek mailing list
>     > zeek at zeek.org
>     > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek



More information about the Zeek mailing list