[Bro-Dev] Underflow Considered Harmful

[Jim Mellander]

Or: How 13 billion became 1.844674e+19 before becoming 0.

After sending amounts totaling over 13 billion thru Sumstats, a value of 0
came out the other end of the sausage factory, but only for one specific
data item.  Debugging this required lots of well placed print statements,
and wild speculation on my part as to what possibly could be broken....

The values being thrown in to be summed are incremental differences from a
previous observation, which *should* be zero or a positive number in the
range of several K, so a 'count' variable was used.  However, for some
reason, this value came up negative (or should have) due to (in decreasing
likelihood) logic error in script, one of bro's dark corners, or bro bug.
The reason for the negativity is still TBD.  But, in the world of unsigned
64-bit values (aka bro 'count' variables) there is no negativity, only
positivity, and an unsigned underflow creates a number just below 2**64 ~
1.844674e+19 ....

Well, Sumstats tallies in doubles, and naturally this figure (1.844674e+19)
dominated the total.  In fact, additional increments to this total pushed
the total value to be greater than 2**64 (with loss of precision, as
doubles only keep 53 bits).

In the processing step at the Sumstats epoch, the value was converted back
to a count using the double_to_count() function which the cheatsheet warns
returns 0, if the double value is <0.0, but it actually returns
2**64-double (with a runtime error), and for values > 2**64 it returns 0
with no runtime error  :-(

So, there it is, a value that should have been about 13 billion became
1.844674e+19 and then became 0.

A few suggestions:
1. Conversion routines should saturate at respective minima/maxima of the
type being converted to (possibly with runtime error).

2. Underflow of the 'count' type is almost invariably a bug, and should
trigger a runtime error.  Overflow, similarly, although in practice it
seems much less likely to occur as most scripts are dealing with integers
considerably less than 2**64.  A similar argument could be made for 'int'.
With some operations, it is difficult to detect overflow/underflow, but for
simple add and subtract, it is relatively easy.

3. Documentation to match behavior.

Jim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180820/fa5734e6/attachment.html