[Bro] problem ingesting bro json logs into splunk
gross.b at ghc.org
Thu Jul 14 08:33:57 PDT 2016
We’ve used bro and splunk at our organization for a couple years now. We utilize the Splunk props and transforms configs to ingest the bro log in the format we want or with the additional attributes and aliases.
REGEX = ^#.*
DEST_KEY = queue
FORMAT = nullQueue
DELIMS = "\t"
FIELDS = ts,uid,id.orig_h,id.orig_p,id.resp_h,id.resp_p,proto,service,duration,orig_bytes,resp_bytes,conn_state,local_orig,local_resp,missed_bytes,history,orig_pkts,orig_ip_bytes,resp_pkts,resp_ip_bytes,tunnel_parents,orig_cc,resp_cc,sensorname
REPORT-bro_conn_extract = bro_conn_extractions
TRANSFORMS-sourcetype = remove_hash_comments
SHOULD_LINEMERGE = false
TRUNCATE = 0
KV_MODE = none
MAX_TIMESTAMP_LOOKAHEAD = 20
TIME_FORMAT = %s.%6N
index = bro_conn
sourcetype = bro_conn
_TCP_ROUTING = primary_indexers
From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of philosnef
Sent: Thursday, July 14, 2016 7:24 AM
Cc: bro at bro.org
Subject: Re: [Bro] problem ingesting bro json logs into splunk
There are no 00.log files in Bro, so the automatic generation of the sourcetype bro_00 makes no sense. It does not follow the standard sourcetype pinning that all the other log files generate. find . -name "00*" in the parent logs directory reports zero logs of this type. This only occured when we moved off of Bro standard log format to JSON format.
On Thursday, July 14, 2016 10:14 AM, Brandon Lattin <lattin at umn.edu<mailto:lattin at umn.edu>> wrote:
Do you have the Splunk installed? (https://splunkbase.splunk.com/app/1617/)
The TA will dynamically create sourcetypes based on the log name.
# Dynamic source typing based on log filename
# Match: conn.log, bro.conn.log,
# md5.bro.conn.log, whatever.conn.log
DEST_KEY = MetaData:Sourcetype
SOURCE_KEY = MetaData:Source
REGEX = ([a-zA-Z0-9-]+)(?:\.[0-9-]*)?(?:\.[0-9\:-]*)?\.log
FORMAT = sourcetype::bro_$1
WRITE_META = true
On Thu, Jul 14, 2016 at 8:33 AM, philosnef <philosnef at yahoo.com<mailto:philosnef at yahoo.com>> wrote:
We are getting a spurious sourcetype when ingesting bro json logs into splunk.
Specifically, we are getting a sourcetype of bro_00. There is no log file named this, and the splunkforwarder is just pushing the raw logs for indexing into splunk. There is no massaging of the log data. Anyone know why this sourcetype is popping up?
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
University of Minnesota - University Information Security
GHC Confidentiality Statement
This message and any attached files might contain confidential information protected by federal and state law. The information is intended only for the use of the individual(s) or entities originally named as addressees. The improper disclosure of such information may be subject to civil or criminal penalties. If this message reached you in error, please contact the sender and destroy this message. Disclosing, copying, forwarding, or distributing the information by unauthorized individuals or entities is strictly prohibited by law.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Bro