[Bro] Quick smtp-url-extraction question
Aashish Sharma
asharma at lbl.gov
Thu Aug 14 09:30:26 PDT 2014
Bloomfilter code in bro-2.2 or below has had some hash collision issues.
Matthias's fixes became part of bro-2.3 release (from CHANGE log):
- Switch to double hashing for Bloomfilters for better performance. (Matthias Vallentin)
- Bugfix to use full digest length instead of just one byte for Bloomfilter's universal hash function. Addresses BIT-1140. (Matthias Vallentin)
Please see: https://bro-tracker.atlassian.net/browse/BIT-1140
If you run smtp-embedded-url-bloom.bro in bro-2.2 world, You will see a huge number of false positives for "SMTP_Link_in_EMAIL_Clicked"
smtp-embedded-url.bro has exact same functionality, except that it maintains a table of smtp urls and checks http requests against it. So less efficient on memory. I expire the contents of the table in 12hours thus a little limited on visibility too. But still I'd say the code works quite alright, so if you cannot quite immidiately upgrade to bro-2.3, feel free to use: smtp-embedded-url.bro script.
Hope this helps,
Aashish
On Thu, Aug 14, 2014 at 09:06:35AM -0700, Josh Liburdi wrote:
> Aashish,
>
> I'm curious why you suggested only using the bloom filter version of
> this script in Bro 2.3-- is there a reason one wouldn't want to use it
> in Bro 2.2?
>
> Josh
>
> On Thu, Aug 14, 2014 at 7:30 AM, Aashish Sharma <asharma at lbl.gov> wrote:
> > OK. Here is smtp-url-extraction scripts attached with this email. I apologize for the delays in sending.
> >
> > These scripts have been running for > 1 1/2 years so I can say they are fairly stable and should not cause any issues.
> >
> > 1) Please configure site.bro (attached) as per your site specifics and add it to your site/local.bro file.
> >
> > 2) If you are running bro-2.2 or below please use: smtp-url-extraction.bro
> >
> > 3) if you are running bro-2.3, use smtp-url-extraction-bloom.bro - it uses bloom filters to check against URL's in the http stream. So its less taxing on memory compared to (2).
> >
> > This script should log urls embedded in smtp traffic into a file called smtpurl_links.log. Also there are configuration variables such as suspicious_text_in_url, suspicious_text_in_body etc. You can look into smtp-embedded-url.bro (and -bloom.bro) to see kinds of notices it would generate.
> >
> > This script is part of a bigger smtp suite. I will try to collect other scripts and send those out as well.
> >
> > Please let me know if you have any questions or have issues running these scripts.
> >
> > Thanks,
> > Aashish
> > LBNL
> >
> > On Thu, Aug 14, 2014 at 01:51:30PM +0000, Hosom, Stephen M wrote:
> >>
> >> All,
> >>
> >>
> >> I submitted a pull request last week for this. You could technically grab
> >> the script and run it. Since I’m not part of the Bro team though, I can’t
> >> promise that this will continue to work.
> >>
> >>
> >> [1]https://github.com/bro/bro/pull/10
> >>
> >>
> >> I run a variation of this script in my production environment right now.
> >> Keep in mind that it is normally a bad plan to extend an internal Bro
> >> module. Since there’s a pretty high demand for it, if you’d like to modify
> >> this to not extend the internal SMTP modules and be separate, it is a
> >> relatively short task (about 15 minutes).
> >>
> >>
> >> Lastly, this is provided as-is with no warranty, etc. etc.
> >>
> >>
> >> Thanks,
> >>
> >> Stephen
> >>
> >>
> >> From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Lankau,
> >> John
> >> Sent: Thursday, August 14, 2014 8:58 AM
> >> To: James Lay; bro at bro-ids.org
> >> Subject: Re: [Bro] Quick smtp-url-extraction question
> >>
> >>
> >> Seth,
> >>
> >>
> >> +100
> >>
> >>
> >> I just wanted to add that I think that script that logs SMTP URLs would get
> >> a lot of use in our environment as well. It’s been an elusive data point,
> >> but one we really would like to have. We’ve been having high-level
> >> discussions on how to implement something that does this exact process in
> >> our office, so I’d be very interested in using this script once it’s ready
> >> as well.
> >>
> >>
> >> Thanks!
> >>
> >> --John
> >>
> >>
> >> From: [2]bro-bounces at bro.org [[3]mailto:bro-bounces at bro.org] On Behalf Of
> >> James Lay
> >> Sent: Thursday, August 07, 2014 7:50 PM
> >> To: [4]bro at bro-ids.org
> >> Subject: Re: [Bro] Quick smtp-url-extraction question
> >>
> >>
> >> On Thu, 2014-08-07 at 13:39 -0400, Seth Hall wrote:
> >>
> >> On Aug 7, 2014, at 1:30 PM, James Lay <[5]jlay at slave-tothe-box.net> wrote:
> >>
> >> > I would absolutely love a script that would log urls....we all know that quot
> >> ed-printable and bas364 shenanigans may get missed
> >>
> >> Much of that should be handled automatically by the mime analyzer (I'm not sure
> >> of the limits of that offhand).
> >>
> >> > , but every little bit helps..thanks a bunch Seth.
> >>
> >> I'll see if I can get to it soon.
> >>
> >> .Seth
> >>
> >> --
> >> Seth Hall
> >> International Computer Science Institute
> >> (Bro) because everyone has a network
> >> [6]http://www.bro.org/
> >>
> >>
> >> Thanks again Seth.
> >> James
> >>
> >> References
> >>
> >> 1. https://github.com/bro/bro/pull/10
> >> 2. mailto:bro-bounces at bro.org
> >> 3. mailto:bro-bounces at bro.org
> >> 4. mailto:bro at bro-ids.org
> >> 5. mailto:jlay at slave-tothe-box.net
> >> 6. http://www.bro.org/
> >
> >> _______________________________________________
> >> Bro mailing list
> >> bro at bro-ids.org
> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >
> >
> > --
> > Aashish Sharma (asharma at lbl.gov)
> > Cyber Security,
> > Lawrence Berkeley National Laboratory
> > http://go.lbl.gov/pgp-aashish
> > Office: (510)-495-2680 Cell: (510)-612-7971
> >
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
--
Aashish Sharma (asharma at lbl.gov)
Cyber Security,
Lawrence Berkeley National Laboratory
http://go.lbl.gov/pgp-aashish
Office: (510)-495-2680 Cell: (510)-612-7971
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140814/74f518db/attachment.bin
More information about the Bro
mailing list