[Zeek] Using "dbl" instead of "num" in SumStats

[Hui Lin (Hugo)]

Hi Jim,

I think that I finally got it. The code is correct. But my interpretation
is not. I think whatever calculation that we apply on observation, e.g.,
average, sum, is for the data collected within that epoch only. So 'num'
field is the total number of observation within that period while I record
the accumulated total number of observation by far. Originally, I don't
like it as I think that it will be convenient for me to have statistics on
all data. However, it does give me some benefits. As I am using very
low-end computers and switches for experiments, I can easily tell when the
network becomes stable, e.g., having less packet loss, based on the RTT in
each epoch.

P.S. as I am working as a faculty now and I have included Bro in my
teaching, I think that SumStats is suitable for a class project as well.

Best regards,

Hui Lin



On Mon, May 6, 2019 at 11:42 AM Lin, Hui <hlin33 at illinois.edu> wrote:

> I am afraid that is not a typo. I copy paste from the documentation at
> https://docs.zeek.org/en/stable/frameworks/sumstats.html#examples
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.zeek.org_en_stable_frameworks_sumstats.html-23examples&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=OnWVBZe4J-_YZZ3rt0XzAVzULLWssyCdTFXvun65UbM&m=qDuyNdlB4jcLX7dpS69FP5aAsCJZniVWnnwvDdFnS7E&s=RufbpzuxSET3bF44SzcQ7epNvUSD04uHbbQsnbHW5pM&e=>.
> I think what I wrote is consistent with what you provide, instead I
> directly call Key and Observe constructor for the second and third
> parameters.  I am just using standalone version to analyze a pcap. More
> interestingly, as the periodic epoch call back function print out, the
> "num" field of the epoch result can decrease!
>
> On Mon, May 6, 2019 at 10:49 AM Jim Mellander <jmellander at lbl.gov> wrote:
>
>> Hmmm, that SumStats::observe line doesn't seem quite correct.  Generally,
>> observations are in the form:
>>
>> SumStats::observe("foo", [$host=bar], [$dbl=val]);
>>
>> Assuming what you sent was just a typo,  it would be interesting to know
>> whether the same behavior is seen both in a cluster and standalone, as
>> SumStats uses a different code path for those two cases.  If only the
>> cluster gives a different result (likely less than the manual count), then
>> I would be concerned that that not all cluster results are being received
>> by the manager when it composes the results.
>>
>> Jim
>>
>> On Mon, May 6, 2019 at 9:56 AM Hui Lin (Hugo) <hlin33 at illinois.edu>
>> wrote:
>>
>>> Hi Jim,
>>>
>>> I think 'num' field seemed like what I am looking for. However, when I
>>> tried, it is different from the count that I manually made. Here is the
>>> codes that I used to count. As you can see, what I try to is easy,
>>> whenever, an observation is received, I increase the value of a global
>>> value. However, when I print out through epoch call back function, the
>>> value is different from one in 'num'.
>>>
>>> if (...)
>>>      {
>>>      total_res = total_res + 1;
>>>      SumStats::observe("dnp3 rtt", SumStats::Key(),
>>> SumStats::Observation($dbl=latency));
>>>      }
>>>
>>>
>>> Best,
>>>
>>> Hugo
>>>
>>> On Mon, May 6, 2019 at 9:16 AM Jim Mellander <jmellander at lbl.gov> wrote:
>>>
>>>> Hi Hugo:
>>>>
>>>> <snip>
>>>> May I suggest a few things in SumStats? Maybe I missed something, I
>>>> don't know how to directly obtain the number of data recorded in SumStats,
>>>> so I need to declare another global variable to record that. It will be
>>>> useful that we can directly know how many data are recorded by far. The
>>>> reason that I need the number of records is to calculate the 95% or 99%
>>>> confidence interval. It will be great that we can include them directly in
>>>> SumStats as well.
>>>> <snip>
>>>>
>>>> Each result record returned to epoch_result has a 'num' field, which is
>>>> a count of the number of observations that made up that result - is that
>>>> what you're looking for?  If you're looking for a grand total of
>>>> observations, I suppose they could be totalled up from the result records.
>>>>
>>>> Take care,
>>>>
>>>> JIm
>>>>
>>>>
>>>
>>> --
>>> Hui Lin
>>> Ph.D. Candidate (http://hlin33.web.engr.illinois.edu/)
>>> DEPEND (http://depend.csl.illinois.edu/)
>>> ECE, Uni. of Illinois at Urbana-Champaign
>>>
>>>
>
> --
> Hui Lin
> Ph.D. Candidate (http://hlin33.web.engr.illinois.edu/)
> DEPEND (http://depend.csl.illinois.edu/)
> ECE, Uni. of Illinois at Urbana-Champaign
>
>

-- 
Hui Lin
Ph.D. Candidate (http://hlin33.web.engr.illinois.edu/)
DEPEND (http://depend.csl.illinois.edu/)
ECE, Uni. of Illinois at Urbana-Champaign
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190506/1f189143/attachment.html