[Bro] problem ingesting bro json logs into splunk

philosnef philosnef at yahoo.com
Thu Jul 14 09:16:24 PDT 2016


The problem with the Spunk app is that indexing is occuring at time of ingest. This causes the indices of the Bro data to grow extremely fast. Using json and not the Bro app means that the data is indexed by Splunk, resulting in far smaller indices on the splunk indexing servers. This is specifically why we moved away from TSV and to JSON, since it was nuking disk storage for those indices... 

    On Thursday, July 14, 2016 12:07 PM, Brandon Lattin <lattin at umn.edu> wrote:
 

 The Bro TA is assuming TSV extractions. The move to JSON probably is causing the Splunk auto-sourcetyper to do some funky things.
[source::...bro.*.log]SHOULD_LINEMERGE = falseTRUNCATE = 0MAX_TIMESTAMP_LOOKAHEAD = 20TIME_FORMAT = %s.%6NTRANSFORMS-BroAutoType = BroAutoType, TrashCommentsINDEXED_EXTRACTIONS = TSVFIELD_HEADER_REGEX = ^#fields\t(.*)FIELD_DELIMITER = \tFIELD_QUOTE = \t
On Thu, Jul 14, 2016 at 9:23 AM, philosnef <philosnef at yahoo.com> wrote:

There are no 00.log files in Bro, so the automatic generation of the sourcetype bro_00 makes no sense. It does not follow the standard sourcetype pinning that all the other log files generate. find . -name "00*" in the parent logs directory reports zero logs of this type. This only occured when we moved off of Bro standard log format to JSON format.
 

    On Thursday, July 14, 2016 10:14 AM, Brandon Lattin <lattin at umn.edu> wrote:
 

 Do you have the Splunk installed? (https://splunkbase.splunk.com/app/1617/)
The TA will dynamically create sourcetypes based on the log name.
# Dynamic source typing based on log filename# Match: conn.log, bro.conn.log, # md5.bro.conn.log, whatever.conn.log[BroAutoType]DEST_KEY = MetaData:SourcetypeSOURCE_KEY = MetaData:SourceREGEX = ([a-zA-Z0-9-]+)(?:\.[0-9-]*)?(?:\.[0-9\:-]*)?\.logFORMAT = sourcetype::bro_$1WRITE_META = true

On Thu, Jul 14, 2016 at 8:33 AM, philosnef <philosnef at yahoo.com> wrote:

We are getting a spurious sourcetype when ingesting bro json logs into splunk.
Specifically, we are getting a sourcetype of bro_00. There is no log file named this, and the splunkforwarder is just pushing the raw logs for indexing into splunk. There is no massaging of the log data. Anyone know why this sourcetype is popping up?
_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




-- 
Brandon LattinSecurity Analyst
University of Minnesota - University Information Security
Office: 612-626-6672

   
_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




-- 
Brandon LattinSecurity Analyst
University of Minnesota - University Information Security
Office: 612-626-6672

  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160714/08eaeda9/attachment-0001.html 


More information about the Bro mailing list