[Bro] Quick smtp-url-extraction question

Josh Liburdi liburdi.joshua at gmail.com
Thu Aug 14 09:41:59 PDT 2014 

I didn't realize there were hash collision issues with 2.2 and below--
thanks for the info!

Josh

On Thu, Aug 14, 2014 at 9:30 AM, Aashish Sharma <asharma at lbl.gov> wrote:
> Bloomfilter code in bro-2.2 or below has had some hash collision issues.
>
> Matthias's fixes became part of bro-2.3 release (from CHANGE log):
>
> - Switch to double hashing for Bloomfilters for better performance.  (Matthias Vallentin)
> - Bugfix to use full digest length instead of just one byte for Bloomfilter's universal hash function. Addresses BIT-1140.  (Matthias Vallentin)
>
> Please see: https://bro-tracker.atlassian.net/browse/BIT-1140
>
> If you run smtp-embedded-url-bloom.bro in bro-2.2 world, You will see a huge number of false positives for "SMTP_Link_in_EMAIL_Clicked"
>
> smtp-embedded-url.bro has exact same functionality, except that it maintains a table of smtp urls and checks http requests against it. So less efficient on memory. I expire the contents of the table in 12hours thus a little limited on visibility too. But still I'd say the code works quite alright, so if you cannot quite immidiately upgrade to bro-2.3, feel free to use: smtp-embedded-url.bro script.
>
> Hope this helps,
> Aashish
>
>
> On Thu, Aug 14, 2014 at 09:06:35AM -0700, Josh Liburdi wrote:
>> Aashish,
>>
>> I'm curious why you suggested only using the bloom filter version of
>> this script in Bro 2.3-- is there a reason one wouldn't want to use it
>> in Bro 2.2?
>>
>> Josh
>>
>> On Thu, Aug 14, 2014 at 7:30 AM, Aashish Sharma <asharma at lbl.gov> wrote:
>> > OK. Here is smtp-url-extraction scripts attached with this email. I apologize for the delays in sending.
>> >
>> > These scripts have been running for >  1 1/2 years so I can say they are fairly stable and should not cause any issues.
>> >
>> > 1) Please configure site.bro (attached) as per your site specifics and add it to your site/local.bro file.
>> >
>> > 2) If you are running bro-2.2 or below please use: smtp-url-extraction.bro
>> >
>> > 3) if you are running bro-2.3, use smtp-url-extraction-bloom.bro - it uses bloom filters to check against URL's in the http stream. So its less taxing on memory compared to (2).
>> >
>> > This script should log urls embedded in smtp traffic into a file called smtpurl_links.log. Also there are configuration variables such as suspicious_text_in_url, suspicious_text_in_body etc. You can look into smtp-embedded-url.bro (and -bloom.bro) to see kinds of notices it would generate.
>> >
>> > This script is part of a bigger smtp suite. I will try to collect other scripts and send those out as well.
>> >
>> > Please let me know if you have any questions or have issues running these scripts.
>> >
>> > Thanks,
>> > Aashish
>> > LBNL
>> >
>> > On Thu, Aug 14, 2014 at 01:51:30PM +0000, Hosom, Stephen M wrote:
>> >>
>> >>    All,
>> >>
>> >>
>> >>    I submitted a pull request last week for this. You could technically grab
>> >>    the script and run it. Since I’m not part of the Bro team though, I can’t
>> >>    promise that this will continue to work.
>> >>
>> >>
>> >>    [1]https://github.com/bro/bro/pull/10
>> >>
>> >>
>> >>    I run a variation of this script in my production environment right now.
>> >>    Keep  in mind that it is normally a bad plan to extend an internal Bro
>> >>    module. Since there’s a pretty high demand for it, if you’d like to modify
>> >>    this  to not extend the internal SMTP modules and be separate, it is a
>> >>    relatively short task (about 15 minutes).
>> >>
>> >>
>> >>    Lastly, this is provided as-is with no warranty, etc. etc.
>> >>
>> >>
>> >>    Thanks,
>> >>
>> >>    Stephen
>> >>
>> >>
>> >>    From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Lankau,
>> >>    John
>> >>    Sent: Thursday, August 14, 2014 8:58 AM
>> >>    To: James Lay; bro at bro-ids.org
>> >>    Subject: Re: [Bro] Quick smtp-url-extraction question
>> >>
>> >>
>> >>    Seth,
>> >>
>> >>
>> >>    +100
>> >>
>> >>
>> >>    I just wanted to add that I think that script that logs SMTP URLs would get
>> >>    a lot of use in our environment as well.  It’s been an elusive data point,
>> >>    but  one  we  really would like to have.  We’ve been having high-level
>> >>    discussions on how to implement something that does this exact process in
>> >>    our office, so I’d be very interested in using this script once it’s ready
>> >>    as well.
>> >>
>> >>
>> >>    Thanks!
>> >>
>> >>    --John
>> >>
>> >>
>> >>    From: [2]bro-bounces at bro.org [[3]mailto:bro-bounces at bro.org] On Behalf Of
>> >>    James Lay
>> >>    Sent: Thursday, August 07, 2014 7:50 PM
>> >>    To: [4]bro at bro-ids.org
>> >>    Subject: Re: [Bro] Quick smtp-url-extraction question
>> >>
>> >>
>> >>    On Thu, 2014-08-07 at 13:39 -0400, Seth Hall wrote:
>> >>
>> >> On Aug 7, 2014, at 1:30 PM, James Lay <[5]jlay at slave-tothe-box.net> wrote:
>> >>
>> >> > I would absolutely love a script that would log urls....we all know that quot
>> >> ed-printable and bas364 shenanigans may get missed
>> >>
>> >> Much of that should be handled automatically by the mime analyzer (I'm not sure
>> >>  of the limits of that offhand).
>> >>
>> >> > , but every little bit helps..thanks a bunch Seth.
>> >>
>> >> I'll see if I can get to it soon.
>> >>
>> >>   .Seth
>> >>
>> >> --
>> >> Seth Hall
>> >> International Computer Science Institute
>> >> (Bro) because everyone has a network
>> >> [6]http://www.bro.org/
>> >>
>> >>
>> >>    Thanks again Seth.
>> >>    James
>> >>
>> >> References
>> >>
>> >>    1. https://github.com/bro/bro/pull/10
>> >>    2. mailto:bro-bounces at bro.org
>> >>    3. mailto:bro-bounces at bro.org
>> >>    4. mailto:bro at bro-ids.org
>> >>    5. mailto:jlay at slave-tothe-box.net
>> >>    6. http://www.bro.org/
>> >
>> >> _______________________________________________
>> >> Bro mailing list
>> >> bro at bro-ids.org
>> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> >
>> >
>> > --
>> > Aashish Sharma  (asharma at lbl.gov)
>> > Cyber Security,
>> > Lawrence Berkeley National Laboratory
>> > http://go.lbl.gov/pgp-aashish
>> > Office: (510)-495-2680  Cell: (510)-612-7971
>> >
>> > _______________________________________________
>> > Bro mailing list
>> > bro at bro-ids.org
>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> --
> Aashish Sharma  (asharma at lbl.gov)
> Cyber Security,
> Lawrence Berkeley National Laboratory
> http://go.lbl.gov/pgp-aashish
> Office: (510)-495-2680  Cell: (510)-612-7971

More information about the Bro mailing list