[Bro] intel.log file stops getting generated.

[fatema bannatwala]

I think I know what was causing the intel log not to get generated.
As suggested by Justin, I disabled the scan.bro, and the sensors looks in
pretty stable state
with 35-40% memory usage overall.

I realized (and tested) that when the following lines are enabled in
local.bro, the intel.log stops getting generated after a day or so:
@load frameworks/intel/do_expire
redef Intel::item_expiration = 1day;

And when I comment out the above lines, the intel.log doesn't have any
problem, and starts getting generated.
I have been testing it for past two days and verified, with scan.bro
disabled and sensors in pretty good state in terms of resource usage.

Hence, my hypothesis is, something is not working correctly, that is
causing the intel feeds to expire in 1day and Bro no longer
has valid intel feeds in memory to compare it with traffic and hence
causing no intel logs getting generated.

We are pulling down the feeds every day around 6:45am in morning in the bro
feed dir.
 I was thinking that if the feeds are not getting updated
(i.e if the feeds are same as they were before pulling), then it might
cause all the old feeds (longer than 1 day) to expire and hence
Bro not generating intel.log.
But I compared the old feeds with new feeds, and verified that every day
some IPs get added and some get removed.
Hence every day the feeds get modified and don't remain the same.

I will still try to troubleshoot the issue, but for time being I have
disabled the do_expire script so that intel.log file is generated.

Anyone observed this kind of issue/behavior?



Thanks,
Fatema.

On Wed, Jan 25, 2017 at 6:16 PM, fatema bannatwala <
fatema.bannatwala at gmail.com> wrote:

> Great! glad to know that I am not the only one dealing with this glitch in
> scan-NG :-)
> Now it totally makes sense, as I was thinking that in our case darknet and
> un-allocated subnets are same
> but will have to be careful, as you mentioned, when un-allocated subnets
> can get assigned without us knowing!
>
> I knew that there were two upgraded scan scripts available other than the
> one that gets ship with Bro by default,
> one that you wrote scan.bro, and another scan-NG script.
>
> Hence was thinking to migrate to use one of those, and stumbled across
> this darknet and allocated net defining issue.
> (There was no way I could be able to define the complete list of allocated
> subnets in scan-NG config, without missing anything :-) ,
> at that time I though it's required to define the allocated subnets, as
> there's a comment in scan-config.bro that reads like:
>  "####Important to configure for Landmine detection
> #### if subnet_feed is empty then LandMine detection wont work "
>  hence thought of tweaking that setting to rather define darknet, but
> never got around to it)
>
> Great to know that there's already some code written by you that works
> around this issue!
> Thanks a ton for all the explanation and link to your scan script, great
> help!
>  will go through it and see if I can get it up and running in our cluster
> :-)
>
> Thanks Justin!
>
>
>
> On Wed, Jan 25, 2017 at 5:10 PM, Azoff, Justin S <jazoff at illinois.edu>
> wrote:
>
>>
>> > On Jan 25, 2017, at 4:43 PM, fatema bannatwala <
>> fatema.bannatwala at gmail.com> wrote:
>> >
>> > Alrighty, yeah was looking into how to configure the script according
>> to the environment.
>> > It appears that we have to define the list of allocated subnets in the
>> network,
>> > as landmine works on watching connections which are not in allocated
>> subnets.
>> >
>> > Defining the allocated subnets is a pain, have a whole lot list of
>> subnets that are allocated and
>> > have just couple of subnets that constitute the darknet, hence was
>> tweaking around the scripts to change that setting
>> > from defining allocated subnets to rather defining un-allocated
>> subnets, which is much easier.
>>
>> That part is optional(but extremely useful).  I'm glad you brought this
>> up, the darknet configuration problem is something I've been thinking about
>> how to fix:
>>
>> * Some people define darknet as NOT allocated.
>> * Some people know exactly which subnets are dark.
>>
>> I did write a version of the darknet code that auto-tunes itself based on
>> allocated subnets, it's part of my scan code:
>>
>> https://gist.github.com/JustinAzoff/80f97af4f4fbb91ae26492b919a50434
>>
>> One can let it run for a while, and then do a
>>
>>     broctl print Site::used_address_space manager
>>
>> to dump out what it figures out as active, and then put it in a file that
>> does
>>
>>     @load ./dark-nets
>>     redef Site::used_address_space = {
>>     ...
>>     }
>>
>> It's not perfect but it's a start.  broker with its persistent data store
>> support may be what is needed to make it more useful.
>>
>> The only issue is it doesn't support something like a honey net that does
>> technically exist: the auto tuning code will flag it as an allocated
>> subnet.  I need to work out how it should be overridden in cases like that.
>>
>> Aside from the auto detection the function just comes down to
>>
>>     return (a in local_nets && a !in used_address_space);
>>
>> In your case you want this instead
>>
>>     return (a in dark_address_space);
>>
>> so I think the simplest thing that may possibly work for everyone is
>> something like
>>
>>     global dark_address_space: set[subnet] &redef;
>>
>> and change the is_darknet logic to be
>>
>>
>>     if(|dark_address_space|)
>>         return (a in dark_address_space);
>>     else
>>         return (a in local_nets && a !in used_address_space);
>>
>> Or maybe just
>>
>>     return (a in local_nets && (a in dark_address_space || a !in
>> used_address_space);
>>
>> but I could see a different user wanting this instead:
>>
>>     return (a in local_nets && a in dark_address_space && a !in
>> used_address_space);
>>
>> for the use case of "dark_address_space is my darknet subnets, but
>> something may be allocated without us knowing, so double check!"
>>
>> I haven't quite figured this out yet.. Maybe the answer is that there
>> isn't a one size fits all implementation and I just need to have 4
>> is_darknet functions depending on how people want it to work:
>>
>>
>>     return (a in dark_address_space);  #mode=darknet
>>
>>     return (a in local_nets && a !in used_address_space);
>> #mode=not_allocated
>>
>>     return (a in local_nets && (a in dark_address_space || a !in
>> used_address_space); #mode=darknet_or_not_allocated
>>
>>     return (a in local_nets && a in dark_address_space && a !in
>> used_address_space); #mode=darknet_and_not_allocated
>>
>> actually, now that I finally wrote all this out, I see that it's just the
>> 4 combinations of 2 boolean flags.
>>
>> --
>> - Justin Azoff
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170129/89c8a722/attachment.html