[Bro-Dev] 'async' update and proposal

[Robin Sommer]

Ok, agree that it's best to postpone "async". My gut isn't quite as
skeptical about its safety but I see the argument. (Although then
nobody will be allowed to complain about "when" anymore ;-)

Jon, if you want to think more about context/scoping it would be great
to keep the concurrency aspects of this in mind as well, as this could
eventually get us there, too. For more context here are a couple more
pointers to past ideas that eventually led to that CCS paper I had
pointed to earlier:

     - The scope-based concurrency model was originally described in
       Section 5.1 of this paper:
       http://www.icir.org/robin/papers/cc-multi-core.pdf.

     - I actually e found the old concurrency code from the Bro 1.x
       era). I'll send you a pointer to that. "grep '&scope'" over
       those policy scripts yields the output at the end of this mail.

     - I now also remember that we indeed needed the internal tracking
       of the current context; just relying on event parameters wasn't
       sufficient. For concurrency the most tricky parts are events
       triggered indirectly, like through timers, as they will often
       need to follow the same scheduling constraints as the original
       one (not sure if that applies to "async").

Robin

--------- cut -------------------------------------------------------

superlinear/policy//bro.init:# &scope=<scope>".
superlinear/policy//conn.bro:    # determine_service() runs with &scope=pair.
superlinear/policy//demux.bro:event _demux_conn(id: conn_id, tag: string, otag: string, rtag: string)  &scope=connection(id)
superlinear/policy//dns.bro:                msg: dns_msg, query: string) &scope=connection(c$id)
superlinear/policy//firewall.bro:event report_violation(c: connection, r:rule) &scope=connection(c$id)
superlinear/policy//ftp.bro:event add_to_first_seen_cmds(command: string) &scope=custom(command)
superlinear/policy//hot.bro:event check_hot_event(c: connection, state: count) &scope=connection(c$id)
superlinear/policy//icmp.bro:event update_flow(icmp: icmp_conn, id: count, is_orig: bool, payload: string) &scope=hostpair(icmp$orig_h, icmp$resp_h)
superlinear/policy//interconn.bro:event _remove_from_demuxed_conn(id: conn_id) &scope=connection(id)
superlinear/policy//nfs.bro:event nfs_request_getattr(n: connection, fh: string, attrs: nfs3_attrs) &scope=custom(fh)
superlinear/policy//nfs.bro:event nfs_attempt_getattr(n: connection, status: count, fh: string) &scope=custom(fh)
superlinear/policy//nfs.bro:event nfs_request_fsstat(n: connection, root_fh: string, stat: nfs3_fsstat) &scope=custom(root_fh)
superlinear/policy//nfs.bro:event nfs_attempt_fsstat(n: connection, status: count, root_fh: string) &scope=custom(root_fh)
superlinear/policy//notice-action-filters.bro:event notice_alarm_per_orig_tally(n: notice_info, host: addr) &scope=originator(host)
superlinear/policy//notice.bro:# will run with &scope connection so that we can store notice_tags.
superlinear/policy//notice.bro:event NOTICE_conn(n: notice_info) &scope=connection(n$conn$id)
superlinear/policy//portmapper.bro:event _do_pm_request(r: connection, proc: string, addl: string, log_it: bool) &scope=originator(r$id$orig_h)
superlinear/policy//scan.bro:event do_rpts_check(c: connection, num: count) &scope=hostpair(c$id$orig_h, c$id$resp_h)
superlinear/policy//scan.bro:event check_scan(c: connection, established: bool, reverse: bool) &scope=originator(reverse? c$id$resp_h : c$id$orig_h)
superlinear/policy//scan.bro:event account_tried(c: connection, user: string, passwd: string) &scope=originator(c$id$orig_h)
superlinear/policy//signatures.bro:event _do_signature_match_notice(state: signature_state, msg: string, data: string) &scope=originator(state$conn$id$orig_h)
superlinear/policy//signatures.bro:event _do_count_per_resp(state: signature_state, msg: string, data: string) &scope=responder(state$conn$id$resp_h)
superlinear/policy//signatures.bro:event _check_alarm_once(state: signature_state, msg: string, data: string) &scope=custom(state$id)
superlinear/policy//trw-impl.bro:event add_to_friendly_remotes(a: addr) &scope=originator(a)
superlinear/policy//trw-impl.bro:event check_TRW_scan(c: connection, state: string, reverse: bool) &scope=originator(reverse? c$id$resp_h : c$id$orig_h)
superlinear/policy//weird.bro:event report_weird_conn_once(t: time, name: string, addl: string, c: connection, action: NoticeAction) &scope=custom(name)
superlinear/policy//weird.bro:event net_weird(name: string) &scope=custom(name)
superlinear/policy//worm.bro:global worm_list: table[addr] of count &default=0 &read_expire = 2 days; #&scope=originator;
superlinear/policy//worm.bro:                    &default=0 &read_expire = 2 days &expire_func=expi; # &scope=originator;

-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin


On Tue, Feb 13, 2018 at 11:02 -0600, you wrote:

> On Tue, Feb 13, 2018 at 9:44 AM, Robin Sommer <robin at icir.org> wrote:
> 
> > Sounds like we all like that idea. Now the question is if we want to
> > wait for that to materialize (which will take quite a bit more
> > brainstorming and then implementation, obviously), or if we want to
> > get async in in the current state and then put that on the TODO list?
> > I can see arguments either way, curious what others think.
> 
> Releasing the async feature by itself sounds unsafe.  Due to the "code
> understandability" concern people had, it didn't sound like there was
> a big demand to start using it without a solution to that anyway, and
> releasing async by itself risks that solution falling through or being
> delayed/unusable for an extended period, resulting in "weird code"
> being written in the meantime and possibly never rewritten later on
> because "it still technically works".
> 
> I'd maybe even go about implementing context/scope by branching off
> the async branch, just in case there's details that arise that make it
> less suitable than originally thought and have to go back to the
> drawing board.
> 
> - Jon
> 


-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin