[Bro] Get the license usage down in Splunk when indexing Bro logs

fatema bannatwala fatema.bannatwala at gmail.com
Wed Jul 12 16:48:44 PDT 2017


Hey Mike,

So just out of curiosity, I ran a quick search on the files.log for the
past hour, to see
the top most logged mime-types, and here is the top 12 mime-types in the
file:

2630139 application/pkix-cert
 366285 text/plain
 259828 -
 258732 image/gif
 175465 text/html
 142375 image/jpeg
 116151 application/xml
 103263 text/json
  70691 image/png
  48208 application/ocsp-response
  18720 application/ocsp-request
  16267 application/javascript

The Splunk filter can be easily built to ignore or filter the logs with
these mime-types (of-course, only if you don't
want them in Splunk):

In profs.conf:
[bro_files_sourcetype]
TRANSFORMS-null= bro_files_setnull

In transforms.conf:
[brol_files_setnull]
REGEX =
(application\/pkix-cert|text\/plain|image\/gif|text\/html|image\/jpeg)
DEST_KEY = queue
FORMAT = nullQueue

You can add more mime-types in the above REGEX, to filter and send to null
queue in Splunk.
I added top five, just to give you an idea of how it can be implemented.

Hope this helps. :)

Thanks,
Fatema.

On Wed, Jul 12, 2017 at 4:35 PM, Mike Eriksson <mike at swedishmike.org> wrote:

> Fatema,
>
> Trying to filter out on types in the the files.log as well sounds like a
> great idea.
>
> We're a bit more limited as to what we can do ourselves when it comes to
> cloud Splunk but I'm sure they're more than happy to sell some PS time if
> need be. ;)
>
> Once again - many thanks for a very helpful suggestion.
>
> Cheers, Mike
>
> On Wed, Jul 12, 2017 at 9:14 PM fatema bannatwala <
> fatema.bannatwala at gmail.com> wrote:
>
>> We only do filtering on conn logs, as they are the heaviest (in our
>> environment at least), before indexing it in Splunk.
>> Also, if you are ingesting files.log as well, then you can build some
>> similar filters in props and transforms for the
>> mime-type you can ignore (like plain/text etc), that will also reduce
>> some of the volume indexed by your Splunk cluster.
>> I do not know much about the cloud deployment, hence can't comment on
>> that.
>>
>> Regards,
>> Fatema.
>>
>>
>> On Wed, Jul 12, 2017 at 3:51 PM, Mike Eriksson <mike at swedishmike.org>
>> wrote:
>>
>>> Hi Fatema,
>>>
>>> Thats looks ace - I'll definitely have to have a try at implementing
>>> that. Hopefully we'll be able to get that done even though we're on Cloud
>>> instances.
>>>
>>> Many thanks for this - it's really apprecaited.
>>>
>>> Cheers, Mike
>>>
>>> On Wed, Jul 12, 2017 at 8:43 PM fatema bannatwala <
>>> fatema.bannatwala at gmail.com> wrote:
>>>
>>>> Hi Mike,
>>>>
>>>> We also have something similar for brologs indexing in Splunk.
>>>> What we do currently is to drop all the connections whose history had
>>>> just a "Syn" and nothing else,
>>>> i.e dropping all the tcp connections that were just connection attempts.
>>>>
>>>> And the way we implemented it in Splunk, is with following filter on
>>>> the indexers:
>>>>
>>>> In props.conf:
>>>> [bro_conn_sourcetype]
>>>> TRANSFORMS-null= bro_conn_setnull
>>>>
>>>> In transforms.conf
>>>> [bro_conn_setnull]
>>>> REGEX = \b[S]{1}\b
>>>> DEST_KEY = queue
>>>> FORMAT = nullQueue
>>>>
>>>> Hope this helps.
>>>>
>>>> Thanks,
>>>> Fatema.
>>>>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170712/cc9518e9/attachment.html 


More information about the Bro mailing list