[Bro-Dev] [JIRA] (BIT-1441) Logrotation cannot be set when using path_func

Tue Aug 4 13:59:01 PDT 2015

    [ https://bro-tracker.atlassian.net/browse/BIT-1441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=21500#comment-21500 ] 

Justin Azoff commented on BIT-1441:
-----------------------------------

files on try.bro.org eventually expire, so I uploaded it here so it does not get lost.

> Logrotation cannot be set when using path_func
> ----------------------------------------------
>
>                 Key: BIT-1441
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1441
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: 2.4
>         Environment: SLC6, PF_RING, broctl
>            Reporter: Jan Grashoefer
>         Attachments: path_func_bug.bro
>
>
> I had a problem using Bro's filtering on my Bro cluster (using broctl). I wanted to create separate logfiles in JSON format for some streams. As the file name should include the current date, I specified a path_func. So far everything worked as expected. Then I tried to disable the logrotation for these files by setting interv = 0. Unfortunately this did not work. Setting a fixed path, disabling logrotation worked as intended (see [http://try.bro.org/#/trybro/saved/14143] an example of the code I used).
> I investigated this issue and think, I have discovered a problem. The rotation interval for a writer is determined in CreateWriter in manager.cc (see [https://github.com/bro/bro/blob/2b1cd66f17194a30b90490965cbdffdd71c18c09/src/logging/Manager.cc#L1064]) based on the filter. The filter again is determined by writer and path (I don't understand why the name of the filter is not used but there may be reasons). To see whether the interval is set correctly I added some debug output here. Then I did a test specifying a filter for HTTP using path_func and a filter for CONN using a fixed path.
> On my worker I get the expected output (except the interval seems wrong):
>        {quote}  0.000000/1437813255.656896 [logging] Set interval for 'packet_filter' (filter 'default') to '86400.000000'
>          0.000000/1437813255.658523 [logging] Set interval for 'loaded_scripts' (filter 'default') to '86400.000000'
>          0.000000/1437813255.685123 [logging] Set interval for 'communication' (filter 'default') to '86400.000000'
> 1437813255.644956/1437813255.709181 [logging] Set interval for 'stats' (filter 'default') to '86400.000000'
> 1437813255.644965/1437813255.710468 [logging] Set interval for 'weird' (filter 'default') to '86400.000000'
> 1437813255.822196/1437813255.834760 [logging] Set interval for 'reporter' (filter 'default') to '86400.000000'
> 1437813256.015793/1437813256.027556 [logging] Set interval for 'software' (filter 'default') to '86400.000000'
> 1437813256.015793/1437813256.039455 [logging] Set interval for 'files' (filter 'default') to '86400.000000'
> 1437813256.015793/1437813256.040269 [logging] Set interval for 'http' (filter 'default') to '86400.000000'
> 1437813256.015793/1437813256.040504 [logging] Set interval for '/var/opt/bro/logs-json/http-2015-07-25' (filter 'http_json') to '0.000000'
> 1437813257.512453/1437813257.523782 [logging] Set interval for 'x509' (filter 'default') to '86400.000000'
> 1437813260.645607/1437813260.656385 [logging] Set interval for 'conn' (filter 'default') to '86400.000000'
> 1437813260.645607/1437813260.656526 [logging] Set interval for '/var/opt/bro/logs-json/conn' (filter 'conn_json') to '0.000000'
> 1437813262.827012/1437813262.839179 [logging] Set interval for 'dns' (filter 'default') to '86400.000000'
> 1437813263.401981/1437813263.411552 [logging] Set interval for 'ssl' (filter 'default') to '86400.000000'
> 1437813293.565530/1437813293.575182 [logging] Set interval for 'kerberos' (filter 'default') to '86400.000000'{quote}
> But on the manager I get the following:
> {quote}1437813085.377826/1437813085.387819 [logging] Set interval for 'loaded_scripts' (filter 'default') to '3600.000000'
> 1437813085.377826/1437813085.400927 [logging] Set interval for 'communication' (filter 'default') to '3600.000000'
> 1437813089.408731/1437813089.409921 [logging] Set interval for 'reporter' (filter '') to '3600.000000'
> 1437813089.410046/1437813089.411141 [logging] Set interval for 'weird' (filter '') to '3600.000000'
> 1437813089.410046/1437813089.411314 [logging] Set interval for 'packet_filter' (filter '') to '3600.000000'
> 1437813089.411802/1437813089.412948 [logging] Set interval for 'stats' (filter '') to '3600.000000'
> 1437813089.444066/1437813089.445155 [logging] Set interval for 'files' (filter '') to '3600.000000'
> 1437813089.453163/1437813089.454249 [logging] Set interval for 'software' (filter '') to '3600.000000'
> 1437813089.472973/1437813089.474123 [logging] Set interval for 'dns' (filter '') to '3600.000000'
> 1437813089.507522/1437813089.508617 [logging] Set default interval for '/var/opt/bro/logs-json/http-2015-07-25' (filter '')
> 1437813089.508759/1437813089.509852 [logging] Set interval for 'http' (filter '') to '3600.000000'
> 1437813089.523751/1437813089.524868 [logging] Set interval for 'x509' (filter '') to '3600.000000',
> 1437813089.983185/1437813089.984342 [logging] Set interval for 'ssl' (filter '') to '3600.000000'
> 1437813093.316215/1437813093.317350 [logging] Set interval for 'ftp' (filter '') to '3600.000000'
> 1437813094.076354/1437813094.077442 [logging] Set interval for 'conn' (filter '') to '3600.000000'
> 1437813094.077580/1437813094.078657 [logging] Set interval for '/var/opt/bro/logs-json/conn' (filter '') to '0.000000'
> 1437813100.949465/1437813100.950567 [logging] Set interval for 'syslog' (filter '') to '3600.000000'{quote}
> On the manager you can see, that for all worker-generated logs the filter is not known and that the interval for my HTTP-JSON log is set to the default value (Note: The instantiating filter is not known because it is not set in the call in SendAllWritersTo - see [https://github.com/bro/bro/blob/2b1cd66f17194a30b90490965cbdffdd71c18c09/src/logging/Manager.cc#L1174]). So why does it work on the worker? Its because the path of the filter is determined and set during the write: The first write triggers determining the path by the filter. Then the writer is created and path of writer and filter match. The writers on the manager seem to be created without a write and therefore the filter cannot be determined.
> At first I tried to fix the issue by using the name of the filter but as seen in the debug output, the name is not set. I also thought about setting the interval using the WriterBackend::WriterInfo, which is passed to CreateWriter and has a field for the interval, but there is also the postprocessor set in the CreateWriter method. Unfortunately I don't understand how logging is distributed between manager and worker in detail, so I do not know how I can fix this issue.

--
This message was sent by Atlassian JIRA
(v6.5-OD-08-001#65007)