[Zeek] State of p0f support
Michał Purzyński
michalpurzynski1 at gmail.com
Mon Jun 17 23:39:35 PDT 2019
What’s needed here is some heuristics. If I see, for example, windows crypto api, BITS, calls to MS services for reporting what usb devices were plugged in, certain DNS lookups - that’s MS.
Apple also has similar services. So does iOS and Android. It’s more an art than a science ;)
JA3 would also be great.
> On Jun 17, 2019, at 11:34 PM, TQ <nothinrandom at gmail.com> wrote:
>
> @Michal,
>
> That's a really good suggestion! Latest p0f is from 2016, so it's not that maintained anyway I guess. The thing I noticed for software.log is that the OS info gets logged only via software calls from apps like Firefox/Chrome/etc; it would be nice to not rely on this.
>
> Thanks,
>
>> On Mon, Jun 17, 2019 at 9:22 PM <zeek-request at zeek.org> wrote:
>> Send Zeek mailing list submissions to
>> zeek at zeek.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>> or, via email, send a message with subject or body 'help' to
>> zeek-request at zeek.org
>>
>> You can reach the person managing the list at
>> zeek-owner at zeek.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of Zeek digest..."
>>
>>
>> Today's Topics:
>>
>> 1. State of p0f support (Robin Sommer)
>> 2. Re: State of p0f support (Micha? Purzy?ski)
>> 3. Re: Hui Lin_DNP3 analyzer not working in current version of
>> zeek (Jon Siwek)
>> 4. Re: Hui Lin_DNP3 analyzer not working in current version of
>> zeek (Hui Lin (Hugo))
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Mon, 17 Jun 2019 13:45:27 -0700
>> From: Robin Sommer <robin at corelight.com>
>> Subject: [Zeek] State of p0f support
>> To: zeek <zeek at zeek.org>
>> Message-ID: <20190617204527.GB61709 at corelight.com>
>> Content-Type: text/plain; charset=us-ascii
>>
>> Looking for some input here.
>>
>> Zeek has provided support for passive OS fingerprinting for a long
>> time through p0f. However, we are using using a very outdated version
>> of the p0f engine, and the signature set is likewise stale (last
>> update from 2011!).
>>
>> Unfortunately p0f has changed quite a bit in meantime, so that it's
>> not easy to upgrade. While we'd certainly be happy to do that if
>> anybody wanted to work on it, for now we are considering to remove the
>> old engine that's currently shipping with Zeek because it doesn't seem
>> to provide much value anymore.
>>
>> Please chime in if that would be a problem for you. Is anybody still
>> relying on the p0f support in Zeek as it is today?
>>
>> Thanks,
>>
>> Robin
>>
>>
>> --
>> Robin Sommer * Corelight, Inc. * robin at corelight.com * www.corelight.com
>>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Mon, 17 Jun 2019 15:05:18 -0700
>> From: Micha? Purzy?ski <michalpurzynski1 at gmail.com>
>> Subject: Re: [Zeek] State of p0f support
>> To: Robin Sommer <robin at corelight.com>
>> Cc: zeek <zeek at zeek.org>
>> Message-ID:
>> <CAJ6bFK1MzSH=fpWAcDQu2ANFTLAGOiicBHA33ZNL=5De2NLQuA at mail.gmail.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>> There is so much data in various logs, like software.log, http.log, SSL,
>> DNS, known_*, x509 and even in the conn.log that recognizing the OS is most
>> of the time trivial. I would rather invest into correlation and build a
>> scoring engine that logs a verdict "based on A, B and C I think this is a
>> Windows 10"
>>
>> On Mon, Jun 17, 2019 at 1:55 PM Robin Sommer <robin at corelight.com> wrote:
>>
>> > Looking for some input here.
>> >
>> > Zeek has provided support for passive OS fingerprinting for a long
>> > time through p0f. However, we are using using a very outdated version
>> > of the p0f engine, and the signature set is likewise stale (last
>> > update from 2011!).
>> >
>> > Unfortunately p0f has changed quite a bit in meantime, so that it's
>> > not easy to upgrade. While we'd certainly be happy to do that if
>> > anybody wanted to work on it, for now we are considering to remove the
>> > old engine that's currently shipping with Zeek because it doesn't seem
>> > to provide much value anymore.
>> >
>> > Please chime in if that would be a problem for you. Is anybody still
>> > relying on the p0f support in Zeek as it is today?
>> >
>> > Thanks,
>> >
>> > Robin
>> >
>> >
>> > --
>> > Robin Sommer * Corelight, Inc. * robin at corelight.com * www.corelight.com
>> > _______________________________________________
>> > Zeek mailing list
>> > zeek at zeek.org
>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>> >
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190617/3f89f3bf/attachment-0001.html
>>
>> ------------------------------
>>
>> Message: 3
>> Date: Mon, 17 Jun 2019 19:46:57 -0700
>> From: Jon Siwek <jsiwek at corelight.com>
>> Subject: Re: [Zeek] Hui Lin_DNP3 analyzer not working in current
>> version of zeek
>> To: "Hui Lin (Hugo)" <hlin33 at illinois.edu>
>> Cc: zeek <zeek at zeek.org>
>> Message-ID:
>> <CAMzgZ0+RHVhvSv5UHVhOz-4cgEg3_km4eDd+NcCic9SR8eEuXg at mail.gmail.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>> Found the difference:
>>
>> This behavior changed in Bro 2.5.4 (really what changed is in BinPAC
>> 0.49, not Bro), but the new parsing behavior is legitimate. The old
>> behavior just caused broken protocol grammars to possibly parse more
>> things than they should have, such as in cases where there wasn't enough
>> data to fill an array. So it appeared to be working for you, but it
>> was not.
>>
>> In this case, the DNP3 protocol grammar we use is either incomplete or
>> needs a further fix. With some debugging for this example pcap, you can
>> see where the parsing is failing:
>>
>> protocol violation, [orig_h=10.0.0.3, orig_p=37147/tcp,
>> resp_h=10.0.0.1, resp_p=20000/tcp], Analyzer::ANALYZER_DNP3_TCP,
>> Binpac exception: binpac exception: out_of_bound:
>> Request_Objects:ojbects: 8 > 0
>> protocol violation, [orig_h=10.0.0.3, orig_p=55021/tcp,
>> resp_h=10.0.0.2, resp_p=20000/tcp], Analyzer::ANALYZER_DNP3_TCP,
>> Binpac exception: binpac exception: out_of_bound:
>> Request_Objects:ojbects: 8 > 0
>>
>> That's here:
>>
>> https://github.com/zeek/zeek/blob/e2dc0092f3a1caea1ebc71e347663e723298fb6b/src/analyzer/protocol/dnp3/dnp3-protocol.pac#L97
>>
>> You can look in the pcap (e.g. Wireshark) and see in the first READ
>> request that there's no objects being sent for us to parse even though
>> our protocol definition is written to expect that. So that protocol
>> violation is legit in the sense that we've defined the protocol in a way
>> that differs from what's being sent on the wire. And a protocol
>> violation in the case of DNP3 disables all further analysis.
>>
>> You maybe understand DNP3 better than me, so please create an issue
>> or pull request if you come up with a fix that improves the DNP3 parser.
>> Attached is a naive patch that seems to generate the same number of
>> requests/responses as before Bro 2.5.4; maybe it helps as a
>> starting point or reference.
>>
>> - Jon
>>
>> On Mon, Jun 17, 2019 at 11:53 AM Hui Lin (Hugo) <hlin33 at illinois.edu> wrote:
>> >
>> > Thanks Jon,
>> >
>> > The version that is working is version 2.5-457. I have attached the sample pcap here.
>> >
>> > Best,
>> >
>> > Hui Lin
>> >
>> > On Mon, Jun 17, 2019 at 10:32 AM Jon Siwek <jsiwek at corelight.com> wrote:
>> >>
>> >> On Sun, Jun 16, 2019 at 3:17 PM Hui Lin (Hugo) <hlin33 at illinois.edu> wrote:
>> >> > Actually for the same pcap, in a version that I git last year, bro works fine by printing all messages. Any idea what happens? If needed, I can provide the pcap for the testing.
>> >>
>> >> Not sure, please either try to debug / explore the diffs, or provide a
>> >> pcap and say which was the last known working version.
>> >>
>> >> - Jon
>> >
>> >
>> >
>> > --
>> > Hui Lin
>> > Ph.D. Candidate (http://hlin33.web.engr.illinois.edu/)
>> > DEPEND (http://depend.csl.illinois.edu/)
>> > ECE, Uni. of Illinois at Urbana-Champaign
>> >
>> -------------- next part --------------
>> A non-text attachment was scrubbed...
>> Name: dnp3-naive.patch
>> Type: application/octet-stream
>> Size: 1229 bytes
>> Desc: not available
>> Url : http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190617/e6d65dd9/attachment-0001.obj
>>
>> ------------------------------
>>
>> Message: 4
>> Date: Mon, 17 Jun 2019 21:17:40 -0700
>> From: "Hui Lin (Hugo)" <hlin33 at illinois.edu>
>> Subject: Re: [Zeek] Hui Lin_DNP3 analyzer not working in current
>> version of zeek
>> To: Jon Siwek <jsiwek at corelight.com>
>> Cc: zeek <zeek at zeek.org>
>> Message-ID:
>> <CAKq214mGdRLKPDZtaXe-w6juvBcpUpmmz6SFYrxgBOfsPwVVQA at mail.gmail.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>> Hi Jon,
>>
>> This is a very confusing part of DNP3 protocol, but only in the request.
>> The "number_of_item" obtained from "object_header" in the request is mainly
>> applied to the response, telling outstation that this is the number of
>> items that you should include in the response. But there are a few
>> exceptions where the request is also using the "number_of_item", especially
>> the control operations, meaning that these are the operations that we would
>> like to apply to the outstation.
>>
>> Back then when I had implemented it, I use a common record
>> type, Request_Data_Object, to hide those differences. In the READ request,
>> there still should be Request_Data_Object[8]. However, according to the
>> value of the function code (which is the input of the Request_Data_Object),
>> the size of each Request_Data_Object becomes 0 if it is READ request. Even
>> though there are 8 objects, but each of them has 0 bytes, so totally there
>> are no data coming. That is why actually there are no more data coming,
>> which is not a mistake. I guess that it is probably how binpac changes the
>> way to handle this type of situation now, making the analyzer fail to work.
>>
>> For me, the workaround should not be difficult (also I am pretty sure that
>> Robin and Seth would like to keep the binpac works as it is now). As there
>> are only a very few exceptions, I just include those exceptions directly
>> there (similar to what you did in the patch), and then default most other
>> requests to empty objects.
>>
>> I probably will do that on July as I am catching a deadline on July 1st. I
>> will contact you privately as I may need you to help me to input a
>> different public key for upload as I have already graduated.
>>
>> Best,
>>
>> Hui Lin
>>
>> On Mon, Jun 17, 2019 at 7:57 PM Jon Siwek <jsiwek at corelight.com> wrote:
>>
>> > Found the difference:
>> >
>> > This behavior changed in Bro 2.5.4 (really what changed is in BinPAC
>> > 0.49, not Bro), but the new parsing behavior is legitimate. The old
>> > behavior just caused broken protocol grammars to possibly parse more
>> > things than they should have, such as in cases where there wasn't enough
>> > data to fill an array. So it appeared to be working for you, but it
>> > was not.
>> >
>> > In this case, the DNP3 protocol grammar we use is either incomplete or
>> > needs a further fix. With some debugging for this example pcap, you can
>> > see where the parsing is failing:
>> >
>> > protocol violation, [orig_h=10.0.0.3, orig_p=37147/tcp,
>> > resp_h=10.0.0.1, resp_p=20000/tcp], Analyzer::ANALYZER_DNP3_TCP,
>> > Binpac exception: binpac exception: out_of_bound:
>> > Request_Objects:ojbects: 8 > 0
>> > protocol violation, [orig_h=10.0.0.3, orig_p=55021/tcp,
>> > resp_h=10.0.0.2, resp_p=20000/tcp], Analyzer::ANALYZER_DNP3_TCP,
>> > Binpac exception: binpac exception: out_of_bound:
>> > Request_Objects:ojbects: 8 > 0
>> >
>> > That's here:
>> >
>> >
>> > https://github.com/zeek/zeek/blob/e2dc0092f3a1caea1ebc71e347663e723298fb6b/src/analyzer/protocol/dnp3/dnp3-protocol.pac#L97
>> >
>> > You can look in the pcap (e.g. Wireshark) and see in the first READ
>> > request that there's no objects being sent for us to parse even though
>> > our protocol definition is written to expect that. So that protocol
>> > violation is legit in the sense that we've defined the protocol in a way
>> > that differs from what's being sent on the wire. And a protocol
>> > violation in the case of DNP3 disables all further analysis.
>> >
>> > You maybe understand DNP3 better than me, so please create an issue
>> > or pull request if you come up with a fix that improves the DNP3 parser.
>> > Attached is a naive patch that seems to generate the same number of
>> > requests/responses as before Bro 2.5.4; maybe it helps as a
>> > starting point or reference.
>> >
>> > - Jon
>> >
>> > On Mon, Jun 17, 2019 at 11:53 AM Hui Lin (Hugo) <hlin33 at illinois.edu>
>> > wrote:
>> > >
>> > > Thanks Jon,
>> > >
>> > > The version that is working is version 2.5-457. I have attached the
>> > sample pcap here.
>> > >
>> > > Best,
>> > >
>> > > Hui Lin
>> > >
>> > > On Mon, Jun 17, 2019 at 10:32 AM Jon Siwek <jsiwek at corelight.com> wrote:
>> > >>
>> > >> On Sun, Jun 16, 2019 at 3:17 PM Hui Lin (Hugo) <hlin33 at illinois.edu>
>> > wrote:
>> > >> > Actually for the same pcap, in a version that I git last year, bro
>> > works fine by printing all messages. Any idea what happens? If needed, I
>> > can provide the pcap for the testing.
>> > >>
>> > >> Not sure, please either try to debug / explore the diffs, or provide a
>> > >> pcap and say which was the last known working version.
>> > >>
>> > >> - Jon
>> > >
>> > >
>> > >
>> > > --
>> > > Hui Lin
>> > > Ph.D. Candidate (http://hlin33.web.engr.illinois.edu/)
>> > > DEPEND (http://depend.csl.illinois.edu/)
>> > > ECE, Uni. of Illinois at Urbana-Champaign
>> > >
>> > _______________________________________________
>> > Zeek mailing list
>> > zeek at zeek.org
>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>
>>
>>
>> --
>> Hui Lin
>> Ph.D. Candidate (http://hlin33.web.engr.illinois.edu/)
>> DEPEND (http://depend.csl.illinois.edu/)
>> ECE, Uni. of Illinois at Urbana-Champaign
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190617/06350065/attachment.html
>>
>> ------------------------------
>>
>> _______________________________________________
>> Zeek mailing list
>> Zeek at zeek.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>>
>>
>> End of Zeek Digest, Vol 158, Issue 22
>> *************************************
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20190617/3755226d/attachment-0001.html
More information about the Zeek
mailing list