[Zeek] intel framework, disabling certain feeds to certain workers

[Munroe Sollog]

We are extending the concept of a bro/zeek cluster.  In addition to our
traditional cluster that analyzes large bandwidth taps, we started
evaluating using additional 'workers' as sensors on servers to
collect targeted data.  For example, on a web proxy we collect web traffic,
on DNS server we collect DNS queries, etc...  We utilize the 'aux_scripts'
feature in nodes.cfg of broctl to define capture filters appropriate for
each service, which reduces load required to run those "sensors".  This
concept has allowed us centrally manage all workers and aggregate data from
many sources to one main pipeline.

In addition, we are ingesting many "security feeds" from many sources.
Currently the cumulative size of all intel data files exceeds 3GB.  The
"traditional" cluster has no problem loading that intel.  However, these
small "sensors" do. A capture filter of, for example:

redef capture_filters += {

    ["dns"] = "port 53"

};

will never match any intel with types: Intel::FILE_NAME, Intel::FILE_HASH,
INTEL::URL.  Allowing a bit more fine-grained control of how workers
operate would allow us to maintain the centralized collection and control
and scale our concept out to other applications without exploding resource
requirements.

Hope this clarifies our use case.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20191114/81b5d632/attachment.html