<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">It seems that there's some inconsistency in SumStats plugin usage and implementation. There appear to be 2 classes of plugins with differing calling mechanisms and action:</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><ol><li>Item to be measured is in the Key, and the measurement is in Observation</li><ol><li>These include Average, Last X Observations, Max, Min, Sample, Standard Deviation, Sum, Unique, Variance</li><ol><li>These are exact measurements.</li><li>Some of these have dependencies: StdDev depends on Variance, which depends on Average<br></li></ol></ol><li>Item to be measured is in Observation, and the measurement is implicitly 1, and the Key is generally null<br></li><ol><li>These include HyperLogLog (number of Unique), TopK (top count)<br></li><ol><li>These are probabilistic data structures.<br></li></ol></ol></ol></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">The Key is not passed to the plugin, but is used to allocate a table that includes, among other things, the processed observations. Both classes call the epoch_result function once per key at the end of the epoch. Since class 2 plugins often use a null key, there is only one call to epoch_result, and a special function is used to extract the results from the probabilistic data structure (<a href="https://www.bro.org/current/exercises/sumstats/sumstats-5.bro">https://www.bro.org/current/exercises/sumstats/sumstats-5.bro</a>). The epoch_finished function is called when all keys have been returned to finish up. This is unneeded with this sort of class 2 plugin, since all the work can be done in the sole call to epoch_result. Multiple keys could be used with class 2 plugins, which allows for groupings (<a href="https://www.bro.org/current/exercises/sumstats/sumstats-4.bro">https://www.bro.org/current/exercises/sumstats/sumstats-4.bro</a>).<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">I have a use case where I want to pass both a key and measurement to a plugin maintaining a probabilistic data store [1]. I don't want to allocate a table for each key, since many/most will not be reflected in the final results. Since the Observation is a record containing both a string & a number, a hack would be to coerce the key to a string, and pass both in the Observation to a class 2 plugin, with a null key - which is what I am doing currently.<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">It would be nice to have a conversation on how to unify these two classes of plugins. A few thoughts on this:</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><ul><li>Pass Key to the plugins - maybe Key could be added to the Observation structure.<br></li><li>Provide a mechanism to *not* allocate the table structure with every new Key (this and the previous can possibly be done with some hackiness with the normalize_key function in the reducer record)</li><li>Some sort of epoch_result factory function that by default just performs the class 1 plugin behavior. For class 2 plugins, the function would feed the results one by one into epoch_result.</li></ul><div><div>Incidentally, I think theres a bug in the observe() function:</div><div><br></div><div>These two lines are run in the loop thru the reducers:</div><div><span style="font-family:monospace,monospace"> if ( r?$normalize_key )<br> key = r$normalize_key(copy(key));</span><br></div><div>which
has the effect of modifying the key for subsequent loops, rather than
just for the one reducer it applies to. The fix is easy and and
obvious....</div><div><br></div></div><div>Jim</div><div><br></div></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">[1] Implementation of algorithms 4&5 (with enhancements) of <a href="https://arxiv.org/pdf/1705.07001.pdf">https://arxiv.org/pdf/1705.07001.pdf</a></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 2, 2018 at 4:44 PM, Jim Mellander <span dir="ltr"><<a href="mailto:jmellander@lbl.gov" target="_blank">jmellander@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Hi all:</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">I'm thinking of writing a SumStats plugin, probably with the initial implementation in bro scriptland, with a re-implementation as BIFs if initial tests successful.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">From examining several plugins, it appears that I need to:</div><ul><li>Add NAME of my plugin as an enum to Calculation</li><li>Add optional tunables to Reducer</li><li>Add my data structure to ResultVal</li><li>In register_observe_plugins, register the function to take an observation.</li><li>In init_result_val_hook, add code to initialize data structure.</li><li>In compose_resultvals_hook, add code to merge multiple data structures</li><li>Create function to extract <div style="font-family:arial,helvetica,sans-serif;display:inline" class="gmail_default">from data structure either at epoch_result, or epoch_finished</div></li></ul><div><div style="font-family:arial,helvetica,sans-serif" class="gmail_default">Any thing else I should be aware of?</div><div style="font-family:arial,helvetica,sans-serif" class="gmail_default"><br></div><div style="font-family:arial,helvetica,sans-serif" class="gmail_default">Thanks in advance,</div><div style="font-family:arial,helvetica,sans-serif" class="gmail_default"><br></div><div style="font-family:arial,helvetica,sans-serif" class="gmail_default">Jim</div><div style="font-family:arial,helvetica,sans-serif" class="gmail_default"></div><br></div><div><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div></div>
</blockquote></div><br></div>