new bro "CURRENT" release - 0.8a70
Vern Paxson
vern at icir.org
Sun Feb 8 14:28:37 PST 2004
An updated "CURRENT" version of Bro is now available from the usual location:
ftp://ftp.ee.lbl.gov/bro-pub-0.8-current.tar.gz
This version has a lot of changes, including a much-expanded Bro home page
at http://www-nrg.ee.lbl.gov/bro.html, including a "wish list" for Bro
development projects at http://www-nrg.ee.lbl.gov/bro-wishlist.html
(suggestions/contributions welcomed!) and a powerful new facility for
post-filtering alerts. I've appended the changes since the last "CURRENT"
version (0.8a58).
Vern
-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
0.8a70 Sun Feb 8 14:19:45 PST 2004
- Bro has a new home page at
http://www-nrg.ee.lbl.gov/bro.html
It includes a "wish list" of Bro development projects:
http://www-nrg.ee.lbl.gov/bro-wishlist.html
- The "match" expression has been completely overhauled (Umesh Shankar).
It now has the syntax:
match EXPR1 using EXPR2
Its semantics are complicated, but it's very powerful (see its use for
alert filtering below). EXPR1 can have any type T. EXPR2 must be of
type "set[R]", where R is a record type. R must have the following fields:
$pred - type is "function(T): bool". This is the predicate
associated with the record. It is passed in EXPR1's
value and returns true or false.
$result - can have any type T'. This is the value to use when
if $pred returns true for EXPR1.
$priority - type must be arithmetic (count, int, double). This
is the priority associated with the match of EXPR1
if $pred returns true.
The way the expression works is that EXPR1 is evaluated yielding a
value V. EXPR2 is then evaluated yielding a set of records whose
type includes the above fields. Bro then spins through each of the
records in the set and tests whether its $pred predicate holds for V.
If so, it records the given $result and the associated $priority.
It then returns for the value of the entire expression the $result
with the highest $priority.
Here's an example. The following script:
global match_stuff = {
[$pred = function(a: count): bool { return a > 5; },
$result = "it's big",
$priority = 2],
[$pred = function(a: count): bool { return a > 15; },
$result = "it's really big",
$priority = 3],
[$pred = function(a: count): bool { return T; },
$result = "default",
$priority = 0],
};
print match 0 using match_stuff;
print match 10 using match_stuff;
print match 20 using match_stuff;
when executed will print:
default
it's big
it's really big
(Note that Bro actually will first evalute $priority before evaluating
$pred, and if it already has a better (higher) priority result, it
will not bother calling $pred.)
- There's a new syntax for designating function values (Umesh Shankar).
It currently only works when associating a function value with a
record field in a record constructor:
[$foo(bad_guy: addr) = { launch_counter_DDOS(bad_guy); return 3.14; }]
is equivalent to:
[$foo = function(bad_guy: addr): double = {
launch_counter_DDOS(bad_guy);
return 3.14;
}]
The goal is to make such functions easier on the eye to express.
The changes are (1) no "function" keywork necessary, (2) no function
return type necessary (note, it is inferred from the "return" statement
in the function body; eventually this will work for all functions, not
just those in record constructors), (3) the '=' sign comes after the ')'
rather than before the keyword "function".
Given this syntax, we can rewrite the initialization of match_stuff
in the example above as:
global match_stuff = {
[$pred(a: count) = { return a > 5; },
$result = "it's big",
$priority = 2],
[$pred(a: count) = { return a > 15; },
$result = "it's really big",
$priority = 3],
[$pred(a: count) = { return T; },
$result = "default",
$priority = 0],
};
- The motivation behind these elaborate new mechanisms is to provide a
powerful and streamlined way to filter alerts. According, alert.bro
now processes any alerts generated via ALERT() through a new global,
alert_policy. alert_policy's type is set[alert_policy_item], where
alert_policy_item is:
type alert_policy_item: record {
result: AlertAction;
pred: function(a: alert_info): bool;
priority: count;
};
The idea is that you specify your alert post-filtering by redef'ing
new elements into alert_policy. For example, here are two post-filtering
rules used at LBL to weed out uninteresting alerts:
# Ignore connections marked as sensitive because they're
# NTP to otherwise-sensitive hosts (first clause) or they happen
# to involve 2766/tcp (Solaris Listen), which happens frequently
# to ftp.ee.lbl.gov if Bro misses the PORT negotiation.
[$pred(a: alert_info) =
{
return a$alert == SensitiveConnection &&
(a$conn$id$resp_p == 123/udp || # NTP
a$msg == /Solaris listen service/);
},
$result = ALERT_FILE,
$priority = 1],
# Ignore sensitive URIs if the request was unsuccessful (code 404,
# or not answered.)
[$pred(a: alert_info) =
{
return a$alert == HTTP::HTTP_SensitiveURI &&
a$msg == /.*((\(404)|(no reply)).*/;
},
$result = ALERT_FILE,
$priority = 1],
These rules are part of:
redef alert_policy += {
... these records and others ...
};
The idea behind them is to demote certain alerts that would ordinarily
be syslog'd (i.e., the associated action is ALERT_LOG_ALWAYS) to instead
just be recorded in the alert.$BRO_ID file. Naturally, there are
many other possibilities. For example:
[$pred(a: alert_info) = {
if ( a$alert == FTP::FTP_Sensitive &&
a$msg == /.*crown_jewels.*/ )
{
system("page_the_duty_officer \"crown jewels theft!\"");
return T;
}
else
return F;
},
$result = ALERT_LOG_ALWAYS,
$priority = 1000],
would run the program page_the_duty_officer with the argument "crown
jewels theft!" if an FTP_Sensitive alert was generated and the log message
included the text "crown_jewels". More generally, post-filtering needn't
just be about deciding on how the alert is logged; the processing can
run programs, update tables, etc., just like any other function call might.
- You can use the new function tally_alert_type in an alert_action_filters
initialization in order to suppress immediate logging of an alert and
instead have Bro generate a summary of how many times the given alert
was seen when it exits. You can use another new function, file_alert,
to specify an alert_action_filters initialization that causes the alerts
to just be written to the alert.$BRO_ID file but not otherwise logged.
For example:
redef alert_action_filters += {
# Just summarize various packet capture glitches.
[[RetransmissionInconsistency, ContentGap, DroppedPackets,
AckAboveHole]] =
tally_alert_type,
[RemoteWorm] = file_alert,
};
would specify that RetransmissionInconsistency (etc.) alerts should just
be reported in the log file (log.$BRO_ID) as a total count, and
RemoteWorm should only be put in the alert.$BRO_ID file, but not
otherwise logged or counted.
You could get the same functionality by writing alert_policy records,
but they're quite a bit bulkier than the above. Note that
alert_action_filters entries take precedence over alert_policy
records, but are evaluated *after* the "match" on alert_policy,
so if you have predicates in alert_policy with side-effects (like the
invocation of page_the_duty_officer in the example above), those
will still happen.
- The alert_info record (which is used in calls to ALERT) now has
slots for some more additional information:
user: string; # can hold an assocaited username
filename: string; # an associated filename
method: string; # associated HTTP method
URL: string; # associated URL
n: count; # any associated count/number/status code
(These are all &optional, so you don't need to specify them if they're
not appropriate.) A number of ALERT's in the default policy scripts
have been changed to include these. The intent is to add more such
information in the future. Ideally, alert_policy records shouldn't
be doing checks like "a$msg == /.*((\(404)|(no reply)).*/" but instead
extracting the status code directly from a field of 'a' (which is an
alert_info record).
- ALERT now fills in the '$id' field in the alert_info record with
the $id of the associated connection, if the caller didn't suppy
a $id but did supply a $conn. Likewise, it will fill in $src with
the $orig_h from $id (after first filling in $id). The net result
is that you can rely on $id and $src being set for any alert that
has an associated connection.
- The HTTP analysis scripts (policy/http*.bro) have been converted to
use the "module" facility, similar to how ftp.bro was converted for
0.8a48. This may require changing some of your own scripts, generally
just to add "HTTP::" qualifiers.
- Now that the variables associated with FTP analysis are part of an
"FTP" module, the "ftp_" prefix associated with:
ftp_excessive_filename_len
ftp_excessive_filename_trunc_len
ftp_guest_ids
ftp_hot_cmds
ftp_hot_files
ftp_hot_guest_files
ftp_ignore_invalid_PORT
ftp_ignore_privileged_PASVs
ftp_log
ftp_skip_hot
has been removed, and these are now called:
excessive_filename_len
excessive_filename_trunc_len
guest_ids
hot_cmds
hot_files
hot_guest_files
ignore_invalid_PORT
ignore_privileged_PASVs
log_file
skip_hot
To get to them from other scripts, you specify, for example,
redef FTP::guest_ids = { .... };
whereas before you had to use:
redef FTP::ftp_guest_ids = { .... };
- The new connection logging format introduced in 0.8a57 is now the
default, unless you redef the new variable "traditional_conn_format"
to be T (Robin Sommer). Connections using unidentified ephemeral
ports now have a service of simply "other" rather than other-XXXX.
The 'U' connection status flag has been removed (unless you're using
traditional_conn_format).
- Tables can now be directly indexed by records, and indexing using records
is no longer interchangeable with using a corresponding list of indices
(Umesh Shankar). This may require adjustments to existing policy
scripts.
- Hostnames such as www.google.com now have type set[addr] rather than
a funky internal list type.
- The new function dump_current_packet(file_name: string) dumps a copy of
the current packet to the file with the given name, appending it if the
file already exists (Robin Sommer). The file is in tcpdump format.
A handy use for this is in an event handler for signature_match(),
to record packets that match given signatures.
- The event new_packet() is invoked for each new packet (Robin Sommer).
It currently doesn't provide the packet contents but soon will in
a fashion similar to secondary-filter.bro.
- "cf -f fmt" specifies a strtime() format. -u specifics UTC time rather
than local time (Mark Delow and Craig Leres). cf now has a man page
(Craig Leres).
- Two new variables, backdoor_ignore_local and backdoor_ignore_remote,
can be used to specify backdoor signatures that should be ignored
if the server is local/remote.
- A bug has been fixed in which a "next" executed in the final iteration
of a for loop would mangle the subsequent processing of the outer
statements (Chema Gonzalez).
- Bug fixes for MIME and Base64 processing (Ruoming Pang).
- pcap.bro now builds its filter in the opposite order (restrict_filters
first), which can improve packet filtering performance (Robin Sommer).
- A bug in &default has been fixed.
- Portability for different pcap_compile_nopcap() calling sequences
(Chema Gonzalez).
- Some tweaks for a minor reduction in memory consumption.
- A memory leak for secondary packet filters has been fixed.
- The localization of error messages (what script line they correspond to)
has been improved.
More information about the Bro
mailing list