<div dir="ltr"><div dir="ltr"><div dir="ltr">We are extending the concept of a bro/zeek cluster. In addition to our traditional cluster that analyzes large bandwidth taps, we started evaluating using additional 'workers' as sensors on servers to collect targeted data. For example, on a web proxy we collect web traffic, on DNS server we collect DNS queries, etc... We utilize the 'aux_scripts' feature in nodes.cfg of broctl to define capture filters appropriate for each service, which reduces load required to run those "sensors". This concept has allowed us centrally manage all workers and aggregate data from many sources to one main pipeline.<div><br></div><div>In addition, we are ingesting many "security feeds" from many sources. Currently the cumulative size of all intel data files exceeds 3GB. The "traditional" cluster has no problem loading that intel. However, these small "sensors" do. A capture filter of, for example:</div><div><br></div><div><p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">redef capture_filters += {</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> ["dns"] = "port 53"</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">};</span></p><br>will never match any intel with types: Intel::FILE_NAME, Intel::FILE_HASH, INTEL::URL. Allowing a bit more fine-grained control of how workers operate would allow us to maintain the centralized collection and control and scale our concept out to other applications without exploding resource requirements.</div><div><br></div><div>Hope this clarifies our use case.</div></div></div></div>