[Bro] Bro Splunk file size and removal interaction

[Azoff, Justin S]

I ran into similar issues a while ago, I now use these settings in limits.conf:



min_batch_size_bytes = 1048576000

[thruput]
maxKBps = 0 # means unlimited


-- 
- Justin Azoff

> On Aug 31, 2016, at 11:23 AM, Collyer, Jeffrey W. (jwc3f) <jwc3f at virginia.edu> wrote:
> 
> So I’m logging my Bro in JSON format on my manager node.  I have Splunk ingesting the log files through the Splunk TA from Github : https://github.com/jahshuah/splunk-ta-bro-json
> 
> Everything is working fine except I’m only getting sporadic http.log entries.  Looking in the Splunk logs, it appears that the http.log file is large enough that Splunk isn’t finished indexing it, when it gets rotated/compressed out and the new 1/2 hour files starts to fill.
> 
> Splunk doesn’t seem to do any file locking(a good thing), but the file goes away before its finished with it.  The machine seems to have plenty of resources, and I’ve turned off the index thruput limit on the splunk heavy forwarder.  So I’m not sure if I can make Splunk go any faster.
> 
> Are there any bro settings that would help here?  I thought about rotating the logs more frequently but if volume is the issue that won’t really help.  Is there a way to have bro not compress/remove the file immediately?
> 
> Or anyone tackled this problem and found a different/splunk solution?
> 
> Jeffrey Collyer
> Information Security Engineer
> University of Virginia
> 434-297-6317
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro