[Bro] problem ingesting bro json logs into splunk

Brandon Lattin lattin at umn.edu
Thu Jul 14 09:38:35 PDT 2016


I would assume your index volume (license usage) is significantly greater
though?

You're right, the raw to indexed ratio is abysmal with TSV (we're
getting 0.29:1).


On Thu, Jul 14, 2016 at 11:16 AM, philosnef <philosnef at yahoo.com> wrote:

> The problem with the Spunk app is that indexing is occuring at time of
> ingest. This causes the indices of the Bro data to grow extremely fast.
> Using json and not the Bro app means that the data is indexed by Splunk,
> resulting in far smaller indices on the splunk indexing servers. This is
> specifically why we moved away from TSV and to JSON, since it was nuking
> disk storage for those indices...
>
>
> On Thursday, July 14, 2016 12:07 PM, Brandon Lattin <lattin at umn.edu>
> wrote:
>
>
> The Bro TA is assuming TSV extractions. The move to JSON probably is
> causing the Splunk auto-sourcetyper to do some funky things.
>
> [source::...bro.*.log]
> SHOULD_LINEMERGE = false
> TRUNCATE = 0
> MAX_TIMESTAMP_LOOKAHEAD = 20
> TIME_FORMAT = %s.%6N
> TRANSFORMS-BroAutoType = BroAutoType, TrashComments
> INDEXED_EXTRACTIONS = TSV
> FIELD_HEADER_REGEX = ^#fields\t(.*)
> FIELD_DELIMITER = \t
> FIELD_QUOTE = \t
>
> On Thu, Jul 14, 2016 at 9:23 AM, philosnef <philosnef at yahoo.com> wrote:
>
> There are no 00.log files in Bro, so the automatic generation of the
> sourcetype bro_00 makes no sense. It does not follow the standard
> sourcetype pinning that all the other log files generate. find . -name
> "00*" in the parent logs directory reports zero logs of this type. This
> only occured when we moved off of Bro standard log format to JSON format.
>
>
> On Thursday, July 14, 2016 10:14 AM, Brandon Lattin <lattin at umn.edu>
> wrote:
>
>
> Do you have the Splunk installed? (https://splunkbase.splunk.com/app/1617/
> )
>
> The TA will dynamically create sourcetypes based on the log name.
>
> # Dynamic source typing based on log filename
> # Match: conn.log, bro.conn.log,
> # md5.bro.conn.log, whatever.conn.log
> [BroAutoType]
> DEST_KEY = MetaData:Sourcetype
> SOURCE_KEY = MetaData:Source
> REGEX = ([a-zA-Z0-9-]+)(?:\.[0-9-]*)?(?:\.[0-9\:-]*)?\.log
> FORMAT = sourcetype::bro_$1
> WRITE_META = true
>
>
> On Thu, Jul 14, 2016 at 8:33 AM, philosnef <philosnef at yahoo.com> wrote:
>
> We are getting a spurious sourcetype when ingesting bro json logs into
> splunk.
>
> Specifically, we are getting a sourcetype of bro_00. There is no log file
> named this, and the splunkforwarder is just pushing the raw logs for
> indexing into splunk. There is no massaging of the log data. Anyone know
> why this sourcetype is popping up?
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
>
>
>
>
> --
> Brandon Lattin
> Security Analyst
> University of Minnesota - University Information Security
> Office: 612-626-6672
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
>
>
>
>
> --
> Brandon Lattin
> Security Analyst
> University of Minnesota - University Information Security
> Office: 612-626-6672
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>



-- 
Brandon Lattin
Security Analyst
University of Minnesota - University Information Security
Office: 612-626-6672
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160714/74ff59f8/attachment-0001.html 


More information about the Bro mailing list