[HILTI] HILTI Digest, Vol 83, Issue 3

Robin Sommer robin at icir.org
Mon Jul 31 07:17:43 PDT 2017


Yeah, and also one in the unit tests: unit/sink.spicy

The key is that the unit itself needs to have the argument declared as
parameter, then then connect should work.

Robin

On Mon, Jul 31, 2017 at 09:42 +0000, you wrote:

> Jennifer,
> 
> You can find an example in the HTTP spicy parser:
> 
> https://github.com/rsmmr/hilti/blob/master/libspicy/parsers/http.spicy
> See lines 80-83; line 157; and lines 175-187.
> 
> I hope this helps.
> 
> Mark
> 
> 
> From: hilti-bounces at ICSI.Berkeley.EDU [mailto:hilti-bounces at ICSI.Berkeley.EDU] On Behalf Of Jennifer Gates
> Sent: Saturday, July 29, 2017 10:09 PM
> To: hilti at ICSI.Berkeley.EDU
> Subject: Re: [HILTI] HILTI Digest, Vol 83, Issue 3
> 
> Is there a good example you can point me to for passing the unit to the sub unit as you suggest? Just adding self in the connect line gives me the following error:
> >>> (self)
> <no location>:: error, too many arguments [pass::spicy::Validator]
> 
> Thank you,
> Jenn
> 
> On Fri, Jul 28, 2017 at 3:00 PM, <hilti-request at icsi.berkeley.edu<mailto:hilti-request at icsi.berkeley.edu>> wrote:
>    Send HILTI mailing list submissions to
>            hilti at mailman.ICSI.Berkeley.EDU<mailto:hilti at mailman.ICSI.Berkeley.EDU>
> 
>    To subscribe or unsubscribe via the World Wide Web, visit
>            http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/hilti
>    or, via email, send a message with subject or body 'help' to
>            hilti-request at mailman.ICSI.Berkeley.EDU<mailto:hilti-request at mailman.ICSI.Berkeley.EDU>
> 
>    You can reach the person managing the list at
>            hilti-owner at mailman.ICSI.Berkeley.EDU<mailto:hilti-owner at mailman.ICSI.Berkeley.EDU>
> 
>    When replying, please edit your Subject line so it is more specific
>    than "Re: Contents of HILTI digest..."
> 
> 
>    Today's Topics:
> 
>       1. Re: Spicy Parsing WebSockets - how to end a list
>          (Fernandez, Mark I)
>       2. Re: Spicy Parsing WebSockets - how to end a list (Jennifer Gates)
>       3. Re: Spicy Parsing WebSockets - how to end a list
>          (Fernandez, Mark I)
>       4. Re: Spicy Parsing WebSockets - how to end a list (Robin Sommer)
> 
> 
>    ----------------------------------------------------------------------
> 
>    Message: 1
>    Date: Thu, 27 Jul 2017 19:09:08 +0000
>    From: "Fernandez, Mark I" <mfernandez at mitre.org<mailto:mfernandez at mitre.org>>
>    Subject: Re: [HILTI] Spicy Parsing WebSockets - how to end a list
>    To: Jennifer Gates <jenngates2004 at gmail.com<mailto:jenngates2004 at gmail.com>>, Robin Sommer
>            <robin at icir.org<mailto:robin at icir.org>>
>    Cc: "hilti at ICSI.Berkeley.EDU<mailto:hilti at ICSI.Berkeley.EDU>" <hilti at ICSI.Berkeley.EDU<mailto:hilti at ICSI.Berkeley.EDU>>
>    Message-ID:
>            <MWHPR09MB1533E7FD664492267457EEE8CFBE0 at MWHPR09MB1533.namprd09.prod.outlook.com<mailto:MWHPR09MB1533E7FD664492267457EEE8CFBE0 at MWHPR09MB1533.namprd09.prod.outlook.com>>
> 
>    Content-Type: text/plain; charset="utf-8"
> 
>    Is the sink executing as you intend?  If not, it could be because when the hook ?on %init? is executed when the routine is initialized, but at that time, the element ?wss_data? has not been parsed yet.  Perhaps try:
> 
>    on wss_data {
>          self.sub.connect(new Sub());
>          self.sub.write(self.wss_data);
>    }
> 
>    Mark
> 
>    From: hilti-bounces at ICSI.Berkeley.EDU<mailto:hilti-bounces at ICSI.Berkeley.EDU> [mailto:hilti-bounces at ICSI.Berkeley.EDU<mailto:hilti-bounces at ICSI.Berkeley.EDU>] On Behalf Of Jennifer Gates
>    Sent: Thursday, July 27, 2017 2:56 PM
>    To: Robin Sommer <robin at icir.org<mailto:robin at icir.org>>
>    Cc: hilti at ICSI.Berkeley.EDU<mailto:hilti at ICSI.Berkeley.EDU>
>    Subject: Re: [HILTI] Spicy Parsing WebSockets - how to end a list
> 
>    Thanks for the response.  Unfortunately, the &eod doesn't work with a list, just with bytes.  Because the end of the connection is simply the end of the last byte of actual data, there is no look-ahead token for the list to work. At least that is what it seems to me.
> 
>    Last night I started down another path. I am trying to use a sink construct.  Here is some of the code:
> 
> 
>    export type WS_Handshake_Success = unit {
> 
>            success     : /^(HTTP|http)/;
> 
>            dvalue      : DataValue;
> 
>            svrheaders     : list<Header> ;
> 
>            end_of_hdrs : /\x0d\x0a\x0d\x0a/;
> 
>            wss_data : bytes &eod &convert=terminate($$) &transient -> self.sub;
> 
>            on %init {
> 
>                    self.sub.connect(new Sub());
> 
>            }
>            var sub: sink;
>    };
> 
> 
>    export type Sub = unit {
> 
>    ws_msgs : list<WS_Message>;
> 
>        : DataEnd;
> 
>    };
> 
>    The terminate function appends a byte sequence (constant DataEnd) to the raw bytes that I can then use to identify the end of the data when parsing the messages in the Sub unit. That way the WS_Messages are properly parsed within the Sub unit.
> 
>    The only piece that I'm working on now is how to put that parsed data back into the original WS_Handshake unit. Right now a "print self" statement of the WS_Handshake just lists sub=<sink>. To get the individual list items to show I need to print the sub unit.
> 
>    Is there a way to make the Sub unit a part of the WS_Handshake unit?
> 
>    Thank you,
>    Jenn
> 
> 
> 
> 
>    -------------- next part --------------
>    An HTML attachment was scrubbed...
>    URL: http://mailman.ICSI.Berkeley.EDU/pipermail/hilti/attachments/20170727/c35009df/attachment-0001.html
> 
>    ------------------------------
> 
>    Message: 2
>    Date: Thu, 27 Jul 2017 15:23:16 -0400
>    From: Jennifer Gates <jenngates2004 at gmail.com<mailto:jenngates2004 at gmail.com>>
>    Subject: Re: [HILTI] Spicy Parsing WebSockets - how to end a list
>    To: "Fernandez, Mark I" <mfernandez at mitre.org<mailto:mfernandez at mitre.org>>
>    Cc: "hilti at ICSI.Berkeley.EDU<mailto:hilti at ICSI.Berkeley.EDU>" <hilti at ICSI.Berkeley.EDU<mailto:hilti at ICSI.Berkeley.EDU>>
>    Message-ID:
>            <CAMV56f0hT2-j3TskcZvrSTUQ4RfpVLX1jNcwQAh_VBoOX7hGDw at mail.gmail.com<mailto:CAMV56f0hT2-j3TskcZvrSTUQ4RfpVLX1jNcwQAh_VBoOX7hGDw at mail.gmail.com>>
>    Content-Type: text/plain; charset="utf-8"
> 
>    Hi,
> 
>    I thought the sink would still keep the sub data as part of the original
>    WS_Handshake unit and not a separate new unit. Otherwise, it does appear to
>    be doing what I intended. Changing my code to how you suggested produces
>    the same results.
> 
>    I guess I'm just wondering if there is a way to have the sink data come
>    back to the WS_Handshake unit after it is parsed by the Sub and WS_Messages
>    units? (I'm thinking it can't since it is already parsed.) Or is there away
>    to add the sub unit's ws_msgs field as a variable within the WS_Handshake
>    unit, or somehow append the sub unit to the WS_Handshake unit?
> 
>    Thank you,
>    Jenn
> 
>    Jenn
>    -------------- next part --------------
>    An HTML attachment was scrubbed...
>    URL: http://mailman.ICSI.Berkeley.EDU/pipermail/hilti/attachments/20170727/19d2a046/attachment-0001.html
> 
>    ------------------------------
> 
>    Message: 3
>    Date: Thu, 27 Jul 2017 19:54:52 +0000
>    From: "Fernandez, Mark I" <mfernandez at mitre.org<mailto:mfernandez at mitre.org>>
>    Subject: Re: [HILTI] Spicy Parsing WebSockets - how to end a list
>    To: Jennifer Gates <jenngates2004 at gmail.com<mailto:jenngates2004 at gmail.com>>
>    Cc: "hilti at ICSI.Berkeley.EDU<mailto:hilti at ICSI.Berkeley.EDU>" <hilti at ICSI.Berkeley.EDU<mailto:hilti at ICSI.Berkeley.EDU>>
>    Message-ID:
>            <MWHPR09MB15339E3878A0186204FD0655CFBE0 at MWHPR09MB1533.namprd09.prod.outlook.com<mailto:MWHPR09MB15339E3878A0186204FD0655CFBE0 at MWHPR09MB1533.namprd09.prod.outlook.com>>
> 
>    Content-Type: text/plain; charset="utf-8"
> 
>    >> if there is a way to have the sink data come back to the WS_Handshake
>    >> unit after it is parsed by the Sub and WS_Messages units? (I'm thinking
>    >> it can't since it is already parsed.)
> 
>    I encountered this dilemma too, but did not find a way around it.  In WS_Handhake, you can reference self.wss_data, but it will be just a byte array, will not have structure or format to it.  That is my experience, but I am new to spicy, don?t know all the tips or tricks.
> 
>    Mark
> 
>    From: Jennifer Gates [mailto:jenngates2004 at gmail.com<mailto:jenngates2004 at gmail.com>]
>    Sent: Thursday, July 27, 2017 3:23 PM
>    To: Fernandez, Mark I <mfernandez at mitre.org<mailto:mfernandez at mitre.org>>
>    Cc: hilti at ICSI.Berkeley.EDU<mailto:hilti at ICSI.Berkeley.EDU>
>    Subject: Re: [HILTI] Spicy Parsing WebSockets - how to end a list
> 
>    Hi,
> 
>    I thought the sink would still keep the sub data as part of the original WS_Handshake unit and not a separate new unit. Otherwise, it does appear to be doing what I intended. Changing my code to how you suggested produces the same results.
> 
>    I guess I'm just wondering if there is a way to have the sink data come back to the WS_Handshake unit after it is parsed by the Sub and WS_Messages units? (I'm thinking it can't since it is already parsed.) Or is there away to add the sub unit's ws_msgs field as a variable within the WS_Handshake unit, or somehow append the sub unit to the WS_Handshake unit?
> 
>    Thank you,
>    Jenn
> 
>    Jenn
>    -------------- next part --------------
>    An HTML attachment was scrubbed...
>    URL: http://mailman.ICSI.Berkeley.EDU/pipermail/hilti/attachments/20170727/8b9e7e78/attachment-0001.html
> 
>    ------------------------------
> 
>    Message: 4
>    Date: Fri, 28 Jul 2017 08:21:06 -0700
>    From: Robin Sommer <robin at icir.org<mailto:robin at icir.org>>
>    Subject: Re: [HILTI] Spicy Parsing WebSockets - how to end a list
>    To: Jennifer Gates <jenngates2004 at gmail.com<mailto:jenngates2004 at gmail.com>>,   "Fernandez, Mark I"
>            <mfernandez at mitre.org<mailto:mfernandez at mitre.org>>
>    Cc: hilti at ICSI.Berkeley.EDU<mailto:hilti at ICSI.Berkeley.EDU>
>    Message-ID: <20170728152106.GQ52171 at icir.org<mailto:20170728152106.GQ52171 at icir.org>>
>    Content-Type: text/plain; charset=us-ascii
> 
> 
> 
>    On Thu, Jul 27, 2017 at 14:55 -0400, Jennifer wrote:
> 
>    > Thanks for the response.  Unfortunately, the &eod doesn't work with a list,
>    > just with bytes.
> 
>    Ah, of course, that's right for the attribute. However, thinking about
>    it, lists should actually be implicitly terminated by the end of the
>    input, even without any attribute. There's a test for that in
>    tests/spicy/list/eof.spicy:
> 
>        module Test;
> 
>        export type test = unit {
>            : list<b> &transient;
>        };
> 
>        type b = unit {
>            a: uint8 {
>                print self.a;
>            }
>        };
> 
>    # echo -n 1234 | spicy-driver tests/spicy/list/eof.spicy
>    49
>    50
>    51
>    52
> 
>    (The &transient doesn't make a difference.)
> 
>    Are you getting an error message about a missing look-ahead token when
>    trying this with your list? If you send me the full Spicy code, I'll
>    see if I can figure out why, it might be that the parser generator
>    doesn't quite recognize the situation.
> 
>    > Last night I started down another path. I am trying to use a sink
>    > construct.
> 
>    Nice idea!
> 
>    > Is there a way to make the Sub unit a part of the WS_Handshake unit?
> 
>    If you pass the handshake unit to the new Sub unit
>    ("self.sub.connect(new Sub(self))"), you can then use that inside Sub
>    to assign to variables stored inside the handshake unit, and bring the
>    information back that way. Does that help?
> 
>    Robin
> 
>    --
>    Robin Sommer * ICSI/LBNL * robin at icir.org<mailto:robin at icir.org> * www.icir.org/robin<http://www.icir.org/robin>
> 
> 
>    ------------------------------
> 
>    _______________________________________________
>    HILTI mailing list
>    HILTI at mailman.ICSI.Berkeley.EDU<mailto:HILTI at mailman.ICSI.Berkeley.EDU>
>    http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/hilti
> 
> 
>    End of HILTI Digest, Vol 83, Issue 3
>    ************************************
> 
> 
> 
> 
>    --
>    Jenn
> 
>    ---------------------------------------------------
>    email: jenngates2004 at gmail.com<mailto:jenngates2004 at gmail.com>
>    blog: https://collateralmeaning.blogspot.com
>    twitter: @jgsecure
> 

> _______________________________________________
> HILTI mailing list
> HILTI at mailman.ICSI.Berkeley.EDU
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/hilti



-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin


More information about the HILTI mailing list