[Bro] Data truncated in signature_match and misc questions
Lorenzo Cavallaro
sullivan at cs.ucsb.edu
Fri May 30 16:39:12 PDT 2008
Hi Robin,
On Fri, May 30, 2008 at 01:49:24PM -0700, Robin Sommer wrote:
> (I think I lost track which of these are already solved (if any) so
> I'll just repsonse briefly. Please ask again for the points which
> aren't yet clear. Also, when having several independent questions,
> it's usually easier to keep track if you mail them to list
> separately.)
Yes, you are right... that would have been much better, my bad :-\
I suppose it's too late for this "thread", tho (If not, let me know
and I'll split the questions in different emails ;-))
This is again a long email but I'm trying to give as much information
as possible to allow you to reproduce what I got...
> > Stream reassembly is very important for me, but I suppose Bro
> > takes care of it when matching against signatures.
>
> That's right.
>
> > that's where things got weird (I would have preferred to leverage
> > on signature_match events than instead of digging into the
> > policy).
>
> (Not sure what you're referring to here.)
I meant I would have preferred to go for a signature-matching
approach that reassembles streams for me, than to go for a solution
which requires me to keep track of streams (with tcp_contents), for
instance. However, as I read at the end of this email, maybe
tcp_contents would do as well without requiring extra work to do...
> > [+] signature_match(sig-AAAA) called
> > payload length: 153
> > payload (first 400 bytes): HTTP/1.1 404 Not Found^M^JDate: Wed, 28
> > May 2008 22:07:28 GMT^M^JServer: Apache^M^JContent-Length:
> > 270^M^JConnection: close^M^JContent-Type: text/html...
> >
> > I don't see any AAAA in there... even if that's the payload which
>
> Where in the packet stream is the AAAA (according to tcpdump)?
>
> Generally, the data passed to the signature_match event is the last
> chunk of bytes which triggered the signature match. This might
> include only parts of the text which matches the pattern and in some
> cases even none at all (when other conditions are involved as well;
> I wouldn't expect this with your signature).
>
> As Vern wrote, it's easiest to track down the specifics of what
> you're seeing if you could send a small trace file along with the
> signature.
Here's a summary:
+ HTTP request (I know, nc would have probably been cleaner,
telnet output omitted)
$ telnet security.dico.unimi.it 80
GET /AAAA HTTP/1.0
<nl>
<nl>
+ packet trace, herein attached (trace.out) has been obtained as
# tcpdump -s 1500 -w trace.out -ni eth0 tcp and host security.dico.unimi.it
+ the trace contains 13 TCP segments and the pattern AAAA appears
in the 4th,5th segments (request) and in the 11th (response).
+ the Bro signature is stored in test.sig and (as in the previous
email), looks like the following:
signature test-AAAA
{
event "sig-AAAA"
payload /.*A{4}/
}
pls, note that I tried also with the pattern /.*AAAA/ -- The
reason for this is given ahead when providing an example of a
regex pattern which doesn't work properly (as expected) with a
Bro built-in function (e.g., gsub).
+ Bro (v1.2.1) has been launched as:
# bro -r trace.out -s test.sig hostname
where hostname is the file which loads all the relevant Bro
policy scripts. It loads test-sig.bro as well which implements
the signature_match event handler:
@load alarm
@load conn
@load adu
@load signatures
#@load print-filter
redef capture_filters = { };
redef restrict_filters = { ["idea"] = "host security.dico.unimi.it" };
redef restrict_filters += { ["tcp"] = "tcp" };
event signature_match(state: signature_state, msg: string, data: string)
{
print fmt("[+] signature_match(%s) called", msg);
print fmt("[+] payload length: %d", byte_len(data));
print fmt("[+] payload (retr w/ sub_bytes(data, 0, 400)):\n%s", sub_bytes(data, 0, 400));
}
+ Bro gives the following output:
[+] signature_match(sig-AAAA) called
[+] payload length: 148
[+] payload (retr w/ sub_bytes(data, 0, 400)):
HTTP/1.1 404 Not Found^M^JDate: Fri, 30 May 2008 22:20:38 GMT^M^JServer: Apache/1.3.34 (Debian) PHP/4.4.4-8+etch4 mod_ssl/2.8.25 OpenSSL/0.9.8c^M...
So, at the end:
+ The data returned by the matched signature doesn't contain the
signature itself.
+ The signature is not triggered for the request I made. The
request has been issued by the same host where Bro is running on. Of
course, I'm able to get and see everything with tcpdump and
wireshark. Just for completeness, I'm running a GNU/Debian
Linux testing system, libpcap version 0.9.8-3 (debian package),
tcpdump version 3.9.8 (debian package), wireshark version 1.0.0
(debian package).
> > The point is that I'd like to extract any matching pattern from the
> > payload which triggered the signature. Once the pattern is extracted
> > I'd have to iterate over each element of the string do something.
>
> That's not really possible without building further script-level
> infrastructure yourself. The problem here is that Bro does not
> buffer the connection's payload internally so when you get a
> signature match, you don't have access to any earlier data. You'd
> need to do this buffering yourself but it depends on the specifics
> of your application whether that it feasible.
I know that I cannot automatically extract a pattern by using the
signature-matching approach. That I've to do it on my own by writing
a Bro policy script. I did that and it works. It means that I'm able
to extract the pattern I want from a given payload Bro passes me. Of
course, if Bro doesn't give me this pattern, no extraction is
possible (see previous points).
> > to check whether the metacharacters {} worked properly or not. It
> > turned out they are ok here (signatures) but they don't work, for
> > instance, with gsub.
>
> Again, a trace and sample script would be good which demonstrates
> the gsub problem.
Same scenario as above. Signature mechanism turned off. I'm loading
the following Bro policy script which uses ADU (note: if I got it
right, ADU should work only with HTTP-based communication. Although
this is not my final goal -- and I'll go with tcp_contents or with
the signature approach -- the example has been tested on an
HTTP-based communication). Script follows:
@load adu
#@load print-filter
redef tcp_content_deliver_all_orig = T;
redef tcp_content_deliver_all_resp = T;
redef adu::adu_max_size = 20000;
redef capture_filters = { };
redef restrict_filters = { ["idea"] = "host security.dico.unimi.it" };
function do_handle_adu(ename: string, c: connection, a: adu::adu_state)
{
local candidate = /(.|\n|\r)*A{4,}/;
if ( |a$adu| > 0 ) {
local substrtmp: string;
local tmp: string;
if ( candidate in a$adu ) {
print fmt("[+] %s: candidate in a$adu", ename);
print fmt("[+] orig payload:\n%s", a$adu);
tmp = gsub(a$adu, /[^A]{4,}/, " ");
print fmt("[+] tmp ([^A]{4,} regex): %s\n", tmp);
tmp = gsub(a$adu, /[^A]+/, " ");
print fmt("[+] tmp: ([^A]+ regex) %s\n", tmp);
tmp = gsub(a$adu, /[^AAAA]/, " ");
print fmt("[+] tmp: ([^AAAA] regex) %s\n", tmp);
}
}
}
event adu_tx(c: connection, a: adu::adu_state)
{
do_handle_adu("adu_tx", c, a);
}
event adu_rx(c: connection, a: adu::adu_state)
{
do_handle_adu("adu_rx", c, a);
}
The output is attached in the file test-adu.output. While I do
understand the last output ([^AAAA] regex) and the second one
([^A]+), I don't quite understand the first one ([^A]{4,}) as I would
expect everything but the four consecutive A to be substituted with
spaces. I might have messed up with regex, tho...
> > 2. Does tcp_contents reassembles flows (I don't think so)? I'd use
>
> It does (though see the options dpd_* in policy/bro.init for
> specifics of when Bro reassembles streams).
Cool, thanks. So, would handling tcp_contents be enough (given the
right dpd_* tuning) if I'm just interested in looking for any sequence
of, say, AAAA in the a TCP stream?
> > 3. I'm not able to see packets that are generated by the same host
> > Bro is running on. Is this a normal behavior (performance tuning)?
> > If so, is there a way to disable it just for testing purposes?
>
> That's as OS issue. Iirc, you indeed don't see packets generated
> locally on some OSs, though I don't remember the details here. You
> can check with tcpdump whether libpcap applications like Bro see the
> packets.
I have no issues at all when using other libpcap-based application on
my system (GNU/Debian testing Linux -- details about versions are
given above).
TIA, bye
Lorenzo
--
Lorenzo `Gigi Sullivan' Cavallaro <sullivan at cs.ucsb.edu>
GPG key at http://security.dico.unimi.it/~sullivan/sullivan.asc
Until I loved, life had no beauty;
I did not know I lived until I had loved. (Theodor Korner)
See the reality in your eyes, when the hate makes you blind. (A.H.X)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: trace.out
Type: application/octet-stream
Size: 1697 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20080530/33a4a292/attachment.obj
-------------- next part --------------
[+] adu_rx: candidate in a$adu
[+] orig payload:
HTTP/1.1 404 Not Found^M^JDate: Fri, 30 May 2008 22:20:38 GMT^M^JServer: Apache/1.3.34 (Debian) PHP/4.4.4-8+etch4 mod_ssl/2.8.25 OpenSSL/0.9.8c^M^JConnection: close^M^JContent-Type: text/html; charset=iso-8859-1^M^J^M^J<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">^J<HTML><HEAD>^J<TITLE>404 Not Found</TITLE>^J</HEAD><BODY>^J<H1>Not Found</H1>^JThe requested URL /AAAA was not found on this server.<P>^J<HR>^J<ADDRESS>Apache/1.3.34 Server at <A HREF="mailto:webmaster at cert-it.dico.unimi.it">idea.sec.dico.unimi.it</A> Port 80</ADDRESS>^J</BODY></HTML>^J
[+] tmp ([^A]{4,} regex): A A A AAAA A A A A A
[+] tmp: ([^A]+ regex) A A A AAAA A A A A A
[+] tmp: ([^AAAA] regex) A A A AAAA A A A A A
More information about the Bro
mailing list