[Bro] Get the license usage down in Splunk when indexing Bro logs

[Mike Eriksson]

Fatema,

Many thanks yet again.

Just for your comparison - here's the top 12 from our current files.log,
which is about 50 minutes old and about to rotate out.

1271142 application/pkix-cert
 182904 text/plain
  54187 text/json
  49992 application/xml
  46694 text/html
  30609 -
  19430 image/png
   9421 image/jpeg
   4778 image/gif
   3664 application/x-dosexec
   3049 text/ini
   1304 application/vnd.ms-cab-compressed

Getting rid of just the 'application/pkix-cert' would probably save us lots
of license here so your tip is really good.

Cheers, Mike

On Thu, Jul 13, 2017 at 12:48 AM fatema bannatwala <
fatema.bannatwala at gmail.com> wrote:

> Hey Mike,
>
> So just out of curiosity, I ran a quick search on the files.log for the
> past hour, to see
> the top most logged mime-types, and here is the top 12 mime-types in the
> file:
>
> 2630139 application/pkix-cert
>  366285 text/plain
>  259828 -
>  258732 image/gif
>  175465 text/html
>  142375 image/jpeg
>  116151 application/xml
>  103263 text/json
>   70691 image/png
>   48208 application/ocsp-response
>   18720 application/ocsp-request
>   16267 application/javascript
>
> The Splunk filter can be easily built to ignore or filter the logs with
> these mime-types (of-course, only if you don't
> want them in Splunk):
>
> In profs.conf:
> [bro_files_sourcetype]
> TRANSFORMS-null= bro_files_setnull
>
> In transforms.conf:
> [brol_files_setnull]
> REGEX =
> (application\/pkix-cert|text\/plain|image\/gif|text\/html|image\/jpeg)
>
> DEST_KEY = queue
> FORMAT = nullQueue
>
> You can add more mime-types in the above REGEX, to filter and send to null
> queue in Splunk.
> I added top five, just to give you an idea of how it can be implemented.
>
> Hope this helps. :)
>
> Thanks,
> Fatema.
>
> On Wed, Jul 12, 2017 at 4:35 PM, Mike Eriksson <mike at swedishmike.org>
> wrote:
>
>> Fatema,
>>
>> Trying to filter out on types in the the files.log as well sounds like a
>> great idea.
>>
>> We're a bit more limited as to what we can do ourselves when it comes to
>> cloud Splunk but I'm sure they're more than happy to sell some PS time if
>> need be. ;)
>>
>> Once again - many thanks for a very helpful suggestion.
>>
>> Cheers, Mike
>>
>> On Wed, Jul 12, 2017 at 9:14 PM fatema bannatwala <
>> fatema.bannatwala at gmail.com> wrote:
>>
>>> We only do filtering on conn logs, as they are the heaviest (in our
>>> environment at least), before indexing it in Splunk.
>>> Also, if you are ingesting files.log as well, then you can build some
>>> similar filters in props and transforms for the
>>> mime-type you can ignore (like plain/text etc), that will also reduce
>>> some of the volume indexed by your Splunk cluster.
>>> I do not know much about the cloud deployment, hence can't comment on
>>> that.
>>>
>>> Regards,
>>> Fatema.
>>>
>>>
>>> On Wed, Jul 12, 2017 at 3:51 PM, Mike Eriksson <mike at swedishmike.org>
>>> wrote:
>>>
>>>> Hi Fatema,
>>>>
>>>> Thats looks ace - I'll definitely have to have a try at implementing
>>>> that. Hopefully we'll be able to get that done even though we're on Cloud
>>>> instances.
>>>>
>>>> Many thanks for this - it's really apprecaited.
>>>>
>>>> Cheers, Mike
>>>>
>>>> On Wed, Jul 12, 2017 at 8:43 PM fatema bannatwala <
>>>> fatema.bannatwala at gmail.com> wrote:
>>>>
>>>>> Hi Mike,
>>>>>
>>>>> We also have something similar for brologs indexing in Splunk.
>>>>> What we do currently is to drop all the connections whose history had
>>>>> just a "Syn" and nothing else,
>>>>> i.e dropping all the tcp connections that were just connection
>>>>> attempts.
>>>>>
>>>>> And the way we implemented it in Splunk, is with following filter on
>>>>> the indexers:
>>>>>
>>>>> In props.conf:
>>>>> [bro_conn_sourcetype]
>>>>> TRANSFORMS-null= bro_conn_setnull
>>>>>
>>>>> In transforms.conf
>>>>> [bro_conn_setnull]
>>>>> REGEX = \b[S]{1}\b
>>>>> DEST_KEY = queue
>>>>> FORMAT = nullQueue
>>>>>
>>>>> Hope this helps.
>>>>>
>>>>> Thanks,
>>>>> Fatema.
>>>>>
>>>>
>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170713/b4454647/attachment-0001.html