[Bro] redef LogExpireInterval with JSON log writer?

[Drew Dixon]

Thank you much Seth- in all honesty I probably didn't dig into either
package enough and just started exploring setting up JSON logging
yesterday, certainly possible that I didn't entirely understand what
json-streaming-logs was doing/solving yet.  I hadn't tested either package,
but just installed and enabled and have been testing your
json-streaming-logs package.

I believe you may be correct that json-streaming-logs does most of what I'm
wanting now that I look closer, however, what I'm still not sure about
though is if there is a way to tell the JSON logs to auto "expire" and be
removed off of disk (not just rotate) at a separate expire interval than
the default tab delimited logs.  So, for example- if I have a retention of
say 15 days ( in broctl.cfg setting LogExpireInterval = 15) of archived
logs for the default tab delimited logs.  I want to be able to tell bro
independently of the broctl.cfg global LogExpireInterval setting value that
I want only all of my json_streaming_* logs to expire/be deleted/removed
off of disk after say 1 day while the normal tab delimited logs still
adhere to the 15 day archive retention.

In other words I do not want the JSON logs to eat up disk space since they
will be getting shipped off box and my cold log retention on on box will be
the archives of tab delimited logs.

I see you're keeping iterations of the json_streaming versions of the logs
around in the event a log shipper process or some process is still attached
to the inode and that the creation of the .1, .2, json logs probably keys
off the custom rotation interval (15 min) from what I can tell, which makes
sense to me.  Aside from that, in my testing I see that json_streaming logs
are in fact being archived along with the default tab delimited logs so I'm
assuming that as it stands now the json_streaming .gz log archives will
stick around on disk just as long as my tab delimited archives unless I
scripted something external of bro to remove them on a daily basis.

If this is all correct and I'm not missing anything else, I'm wondering if
it would be possible for you to do something like I described above for
removing the json_streaming_logs archives from disk more frequently with
your package script?  I think bro cron does this now?  So not 100% certain
how that may affect the plausibility of this, if at all.

Respectfully,

-Drew

On Thu, Mar 15, 2018 at 5:48 PM, Seth Hall <seth at corelight.com> wrote:

>
>
> On 15 Mar 2018, at 15:27, Drew Dixon wrote:
>
> Is a redef-able option for the log expire interval something that might be
>> added in a future version of bro?  Is there a way to do this now that I'm
>> just missing? Is LogExpireInterval only available for
>> broctl/broctl.cfg?
>>
>
> What you set with broctl is just the global filter.  If you look at the
> json-streaming-logs package (link included below), you can see that I'm
> setting a custom rotation interval separately from the global default
> rotation interval.  If you are looking to duplicate logging, you're going
> to be doing something similar to what json-streaming-logs is doing.  I'm
> curious if json-streaming-logs doesn't do what you need to.  It's possible
> that if what you need conceptually fits into that package I could just add
> it there.
>
> From how you described your problem, it sounds like json-streaming-logs
> might already do what you need?
>
> Here's the link to how I'm setting a custom rotation interval for a log
> filter that I referenced above:
>         https://github.com/corelight/json-streaming-logs/blob/master
> /scripts/main.bro#L72
>
>   .Seth
>
> --
> Seth Hall * Corelight, Inc * www.corelight.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180316/394138f8/attachment.html