<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Thanks, Jon:</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">I've decided to split the data (a table of IP addresses with statistics captured over a time period) based on a modulo calculation against the IP address (the important characteristic being that it can be done on the fly without an additional pass thru the table), which with an average distribution of traffic gives relatively equal size buckets, each of which can be processed during a single event, as I described.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">I like the idea of co-routines - it would help to address issues like these in a more natural manner.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Jim</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 5, 2018 at 5:28 PM, Jon Siwek <span dir="ltr"><<a href="mailto:jsiwek@corelight.com" target="_blank">jsiwek@corelight.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Fri, Jan 5, 2018 at 2:19 PM, Jim Mellander <<a href="mailto:jmellander@lbl.gov">jmellander@lbl.gov</a>> wrote:<br>
<br>
> I haven't checked whether my desired behavior works, but since its not<br>
> documented, I wouldn't want to rely on it in any event.<br>
<br>
</span>Yeah, I doubt the example you gave currently works -- it would just<br>
change the local value in the frame without modifying the internal<br>
iterator.<br>
<span class=""><br>
> I would be interested in hearing comments or suggestions on this issue.<br>
<br>
</span>What you want, the ability to split the processing of large data<br>
tables/sets over time, makes sense. I've probably also run into at<br>
least a couple cases where I've been concerned about how long it would<br>
take to iterate over a set/table and process all keys in one go. The<br>
approach that comes to mind for doing that would be adding coroutines.<br>
Robin has some ongoing work with adding better support for async<br>
function calls, and I wonder if the way that's done would make it<br>
pretty simple to add general coroutine support as well. E.g. stuff<br>
could look like:<br>
<br>
event process_stuff()<br>
{<br>
local num_processed = 0;<br>
<br>
for ( local item in foo )<br>
{<br>
process_item(item);<br>
<br>
if ( ++num_processed % 1000 == 0 )<br>
yield; # resume next time events get drained (e.g. next packet)<br>
}<br>
<br>
There could also be other types of yield instructions, like "yield 1<br>
second" or "yield wait_for_my_signal()" which would, respectively,<br>
resume after arbitrary amount of time or a custom function says it<br>
should.<br>
<span class="HOEnZb"><font color="#888888"><br>
- Jon<br>
</font></span></blockquote></div><br></div>