[Bro] problem ingesting bro json logs into splunk

[Gross, Brett]

We’ve used bro and splunk at our organization for a couple years now. We utilize the Splunk props and transforms configs to ingest the bro log in the format we want or with the additional attributes and aliases.

transforms.conf:
[remove_hash_comments]
REGEX = ^#.*
DEST_KEY = queue
FORMAT = nullQueue

[bro_conn_extractions]
DELIMS = "\t"
FIELDS = ts,uid,id.orig_h,id.orig_p,id.resp_h,id.resp_p,proto,service,duration,orig_bytes,resp_bytes,conn_state,local_orig,local_resp,missed_bytes,history,orig_pkts,orig_ip_bytes,resp_pkts,resp_ip_bytes,tunnel_parents,orig_cc,resp_cc,sensorname

Props.conf:
[bro_conn]
REPORT-bro_conn_extract = bro_conn_extractions
TRANSFORMS-sourcetype = remove_hash_comments

SHOULD_LINEMERGE = false
TRUNCATE = 0
KV_MODE = none

MAX_TIMESTAMP_LOOKAHEAD = 20
TIME_FORMAT = %s.%6N

Inputs.conf:
[monitor:///your_log_path/log/bro/conn.log]
index = bro_conn
sourcetype = bro_conn
_TCP_ROUTING = primary_indexers


brett

From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of philosnef
Sent: Thursday, July 14, 2016 7:24 AM
Cc: bro at bro.org
Subject: Re: [Bro] problem ingesting bro json logs into splunk

There are no 00.log files in Bro, so the automatic generation of the sourcetype bro_00 makes no sense. It does not follow the standard sourcetype pinning that all the other log files generate. find . -name "00*" in the parent logs directory reports zero logs of this type. This only occured when we moved off of Bro standard log format to JSON format.

On Thursday, July 14, 2016 10:14 AM, Brandon Lattin <lattin at umn.edu<mailto:lattin at umn.edu>> wrote:

Do you have the Splunk installed? (https://splunkbase.splunk.com/app/1617/)

The TA will dynamically create sourcetypes based on the log name.

# Dynamic source typing based on log filename
# Match: conn.log, bro.conn.log,
# md5.bro.conn.log, whatever.conn.log
[BroAutoType]
DEST_KEY = MetaData:Sourcetype
SOURCE_KEY = MetaData:Source
REGEX = ([a-zA-Z0-9-]+)(?:\.[0-9-]*)?(?:\.[0-9\:-]*)?\.log
FORMAT = sourcetype::bro_$1
WRITE_META = true


On Thu, Jul 14, 2016 at 8:33 AM, philosnef <philosnef at yahoo.com<mailto:philosnef at yahoo.com>> wrote:
We are getting a spurious sourcetype when ingesting bro json logs into splunk.

Specifically, we are getting a sourcetype of bro_00. There is no log file named this, and the splunkforwarder is just pushing the raw logs for indexing into splunk. There is no massaging of the log data. Anyone know why this sourcetype is popping up?

_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro<http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>



--
Brandon Lattin
Security Analyst
University of Minnesota - University Information Security
Office: 612-626-6672


________________________________

GHC Confidentiality Statement

This message and any attached files might contain confidential information protected by federal and state law. The information is intended only for the use of the individual(s) or entities originally named as addressees. The improper disclosure of such information may be subject to civil or criminal penalties. If this message reached you in error, please contact the sender and destroy this message. Disclosing, copying, forwarding, or distributing the information by unauthorized individuals or entities is strictly prohibited by law.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160714/05d86df4/attachment-0001.html