[Bro-Dev] [archive log failure]

[Daniel Thayer]

OK, so the next thing to check is if the archive-log and
make-archive-name scripts are receiving command-line parameters
that are in the expected format.  Maybe add a line like this
to those scripts:

   echo $@ >> /tmp/something.out

I'd suggest also checking if there's anything strange in
the <PREFIX>/spool/manager/stderr.log file.


On 10/3/16 4:04 PM, Aashish Sharma wrote:
>> the <PREFIX>/share/broctl/scripts/archive-log and
>> the <PREFIX>/share/broctl/scripts/post-terminate script.
>
> No modifications on either scripts as far as I can tell:
>
> MD5 (share/broctl/scripts/archive-log) = 0d61804be56f8a61c18c6612bad486c8
> MD5 (share/broctl/scripts/post-terminate) = ad4b56dfcfe8c1796a0a755e37256dda
>
> Aashish
>
> On Mon, Oct 03, 2016 at 03:45:54PM -0500, Daniel Thayer wrote:
>> Your make-archive-name script works for me.
>>
>> The next thing to check is your copy of
>> the <PREFIX>/share/broctl/scripts/archive-log and
>> the <PREFIX>/share/broctl/scripts/post-terminate script.
>> Check if you made any changes to those scripts (a bug in
>> those scripts could potentially run make-archive-name with
>> invalid parameters).
>>
>>
>> On 10/3/16 3:18 PM, Aashish Sharma wrote:
>>> HI Daniel,
>>>
>>>> As for the strange directory names, one possible reason could be your
>>>> make-archive-name script is producing bad output.
>>>
>>> make-archive-name script does correctly archive the logs to 2016-10-03 folder.
>>>
>>> This is the contect of the script:
>>>
>>> $ cat make-archive-name
>>>
>>> name=$1
>>> flavor=$2
>>> opened=$3
>>> closed=$4
>>> host=`hostname -s`
>>>
>>> day=`echo $opened  | awk -F - '{printf "%s-%s-%s", $1, $2, $3}'`
>>> from=`echo $opened | awk -F - '{printf "%s:%s:%s", $4, $5, $6}'`
>>> to=`echo $closed | awk -F - '{printf "%s:%s:%s", $4, $5, $6}'`
>>>
>>> if [ "$closed" != "" ]; then
>>>   echo $day/$name.$host.$day-$from-$to
>>> else
>>>   echo $day/$name.$host.$day-$from-current
>>> fi
>>>
>>> ===
>>>
>>> Hereis output of  20rk-5-8 directory for example:
>>>
>>> ~/logs/20rk-5-8]$ ls -altrh
>>> total 40
>>> -rw-r--r--    1 bro  bro    20B Sep 28 17:39 drop-debug.log.cluster.20rk-5-8-::-17:39:24.gz
>>> -rw-r--r--    1 bro  bro    20B Oct  3 09:40 drop-debug.log.cluster.20rk-5-8-::-09:40:35.gz
>>> drwxr-xr-x  196 bro  bro   6.0k Oct  3 11:54 ..
>>> drwxr-xr-x    2 bro  bro   512B Oct  3 11:54 .
>>> -rw-r--r--    1 bro  bro    20B Oct  3 11:54 drop-debug.log.cluster.20rk-5-8-::-11:54:38.gz
>>>
>>>
>>> Since make-archive-name does archive logs as expected not sure how to address 20rk-5-8 issue. secondly, why would these directories be in ~/logs instead of ../spool/tmp ?
>>>
>>>
>>> Aashish
>>>
>>> On Mon, Oct 03, 2016 at 03:02:14PM -0500, Daniel Thayer wrote:
>>>> Those archive log failure emails are a new feature in version 2.5.
>>>> The only purpose of the emails is to make it easier to notice when
>>>> such an error occurs (i.e., these emails do not indicate a new type
>>>> of error condition).
>>>> Previously, if such a failure occurred, the only way you would know
>>>> is if you noticed missing logs in one of the subdirectories of
>>>> the <PREFIX>/logs/ directory, or if you noticed the presence of
>>>> a new spool/tmp/post-terminate-* directory.
>>>>
>>>> As for the strange directory names, one possible reason could be your
>>>> make-archive-name script is producing bad output.
>>>>
>>>>
>>>>
>>>> On 10/3/16 2:11 PM, Aashish Sharma wrote:
>>>>> I see notifications as following:
>>>>>
>>>>> ----- Forwarded message from Xxxxxxx  -----
>>>>>
>>>>> Date: Mon, 3 Oct 2016 11:54:39 -0700 (PDT)
>>>>> From:
>>>>> To:
>>>>> Subject: [bro-cluster] archive log failure
>>>>>
>>>>> Unable to archive one or more logs in directory:
>>>>> /usr/local/bro/spool/tmp/post-terminate-worker-2016-10-03-09-40-35-36665
>>>>> Check the post-terminate.out file in that directory for any error messages.
>>>>>