new bro "CURRENT" release - 0.8a70

Vern Paxson vern at icir.org
Sun Feb 8 14:28:37 PST 2004


An updated "CURRENT" version of Bro is now available from the usual location:

	ftp://ftp.ee.lbl.gov/bro-pub-0.8-current.tar.gz

This version has a lot of changes, including a much-expanded Bro home page
at http://www-nrg.ee.lbl.gov/bro.html, including a "wish list" for Bro
development projects at http://www-nrg.ee.lbl.gov/bro-wishlist.html
(suggestions/contributions welcomed!) and a powerful new facility for
post-filtering alerts.  I've appended the changes since the last "CURRENT"
version (0.8a58).

                Vern


-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+


0.8a70 Sun Feb  8 14:19:45 PST 2004

- Bro has a new home page at

	http://www-nrg.ee.lbl.gov/bro.html

  It includes a "wish list" of Bro development projects:

	http://www-nrg.ee.lbl.gov/bro-wishlist.html

- The "match" expression has been completely overhauled (Umesh Shankar).
  It now has the syntax:

	match EXPR1 using EXPR2

  Its semantics are complicated, but it's very powerful (see its use for
  alert filtering below).  EXPR1 can have any type T.  EXPR2 must be of
  type "set[R]", where R is a record type.  R must have the following fields:

	$pred	- type is "function(T): bool".  This is the predicate
		  associated with the record.  It is passed in EXPR1's
		  value and returns true or false.

	$result	- can have any type T'.  This is the value to use when
		  if $pred returns true for EXPR1.

	$priority - type must be arithmetic (count, int, double).  This
		  is the priority associated with the match of EXPR1
		  if $pred returns true.
  
  The way the expression works is that EXPR1 is evaluated yielding a
  value V.  EXPR2 is then evaluated yielding a set of records whose
  type includes the above fields.  Bro then spins through each of the
  records in the set and tests whether its $pred predicate holds for V.
  If so, it records the given $result and the associated $priority.
  It then returns for the value of the entire expression the $result
  with the highest $priority.

  Here's an example.  The following script:

	global match_stuff = {
		[$pred = function(a: count): bool { return a > 5; },
		 $result = "it's big",
		 $priority = 2],

		[$pred = function(a: count): bool { return a > 15; },
		 $result = "it's really big",
		 $priority = 3],

		[$pred = function(a: count): bool { return T; },
		 $result = "default",
		 $priority = 0],
	};

	print match 0 using match_stuff;
	print match 10 using match_stuff;
	print match 20 using match_stuff;

  when executed will print:

	default
	it's big
	it's really big

  (Note that Bro actually will first evalute $priority before evaluating
  $pred, and if it already has a better (higher) priority result, it
  will not bother calling $pred.)

- There's a new syntax for designating function values (Umesh Shankar).
  It currently only works when associating a function value with a
  record field in a record constructor:

	[$foo(bad_guy: addr) = { launch_counter_DDOS(bad_guy); return 3.14; }]

  is equivalent to:

	[$foo = function(bad_guy: addr): double = {
			launch_counter_DDOS(bad_guy);
			return 3.14;
	}]

  The goal is to make such functions easier on the eye to express.
  The changes are (1) no "function" keywork necessary, (2) no function
  return type necessary (note, it is inferred from the "return" statement
  in the function body; eventually this will work for all functions, not
  just those in record constructors), (3) the '=' sign comes after the ')'
  rather than before the keyword "function".

  Given this syntax, we can rewrite the initialization of match_stuff
  in the example above as:

	global match_stuff = {
		[$pred(a: count) = { return a > 5; },
		 $result = "it's big",
		 $priority = 2],

		[$pred(a: count) = { return a > 15; },
		 $result = "it's really big",
		 $priority = 3],

		[$pred(a: count) = { return T; },
		 $result = "default",
		 $priority = 0],
	};

- The motivation behind these elaborate new mechanisms is to provide a
  powerful and streamlined way to filter alerts.  According, alert.bro
  now processes any alerts generated via ALERT() through a new global,
  alert_policy.  alert_policy's type is set[alert_policy_item], where
  alert_policy_item is:

	type alert_policy_item: record {
		result: AlertAction;
		pred: function(a: alert_info): bool;
		priority: count;
	};

  The idea is that you specify your alert post-filtering by redef'ing
  new elements into alert_policy.  For example, here are two post-filtering
  rules used at LBL to weed out uninteresting alerts:

	# Ignore connections marked as sensitive because they're
	# NTP to otherwise-sensitive hosts (first clause) or they happen
	# to involve 2766/tcp (Solaris Listen), which happens frequently
	# to ftp.ee.lbl.gov if Bro misses the PORT negotiation.
	[$pred(a: alert_info) =
		{
		return a$alert == SensitiveConnection &&
			(a$conn$id$resp_p == 123/udp || # NTP
			 a$msg == /Solaris listen service/);
		},
	 $result = ALERT_FILE,
	 $priority = 1],


	# Ignore sensitive URIs if the request was unsuccessful (code 404,
	# or not answered.)
	[$pred(a: alert_info) =
		{
		return a$alert == HTTP::HTTP_SensitiveURI &&
			a$msg == /.*((\(404)|(no reply)).*/;
		},
	 $result = ALERT_FILE,
	 $priority = 1],

  These rules are part of:

	redef alert_policy += {
		... these records and others ...
	};

  The idea behind them is to demote certain alerts that would ordinarily
  be syslog'd (i.e., the associated action is ALERT_LOG_ALWAYS) to instead
  just be recorded in the alert.$BRO_ID file.  Naturally, there are
  many other possibilities.  For example:

	[$pred(a: alert_info) = {
		if ( a$alert == FTP::FTP_Sensitive &&
		     a$msg == /.*crown_jewels.*/ )
			{
			system("page_the_duty_officer \"crown jewels theft!\"");
			return T;
			}
		else
			return F;
		},
	$result = ALERT_LOG_ALWAYS,
	$priority = 1000],

  would run the program page_the_duty_officer with the argument "crown
  jewels theft!" if an FTP_Sensitive alert was generated and the log message
  included the text "crown_jewels".  More generally, post-filtering needn't
  just be about deciding on how the alert is logged; the processing can
  run programs, update tables, etc., just like any other function call might.

- You can use the new function tally_alert_type in an alert_action_filters
  initialization in order to suppress immediate logging of an alert and
  instead have Bro generate a summary of how many times the given alert
  was seen when it exits.  You can use another new function, file_alert,
  to specify an alert_action_filters initialization that causes the alerts
  to just be written to the alert.$BRO_ID file but not otherwise logged.

  For example:

	redef alert_action_filters += {
		# Just summarize various packet capture glitches.
		[[RetransmissionInconsistency, ContentGap, DroppedPackets,
		  AckAboveHole]] =
			tally_alert_type,

		[RemoteWorm] = file_alert,
	};

  would specify that RetransmissionInconsistency (etc.) alerts should just
  be reported in the log file (log.$BRO_ID) as a total count, and
  RemoteWorm should only be put in the alert.$BRO_ID file, but not
  otherwise logged or counted.

  You could get the same functionality by writing alert_policy records,
  but they're quite a bit bulkier than the above.  Note that
  alert_action_filters entries take precedence over alert_policy
  records, but are evaluated *after* the "match" on alert_policy,
  so if you have predicates in alert_policy with side-effects (like the
  invocation of page_the_duty_officer in the example above), those
  will still happen.

- The alert_info record (which is used in calls to ALERT) now has
  slots for some more additional information:

	user: string;	# can hold an assocaited username
	filename: string;	# an associated filename
	method: string;	# associated HTTP method
	URL: string;	# associated URL
	n: count;	# any associated count/number/status code

  (These are all &optional, so you don't need to specify them if they're
  not appropriate.)  A number of ALERT's in the default policy scripts
  have been changed to include these.  The intent is to add more such
  information in the future.  Ideally, alert_policy records shouldn't
  be doing checks like "a$msg == /.*((\(404)|(no reply)).*/" but instead
  extracting the status code directly from a field of 'a' (which is an
  alert_info record).

- ALERT now fills in the '$id' field in the alert_info record with
  the $id of the associated connection, if the caller didn't suppy
  a $id but did supply a $conn.  Likewise, it will fill in $src with
  the $orig_h from $id (after first filling in $id).  The net result
  is that you can rely on $id and $src being set for any alert that
  has an associated connection.

- The HTTP analysis scripts (policy/http*.bro) have been converted to
  use the "module" facility, similar to how ftp.bro was converted for
  0.8a48.  This may require changing some of your own scripts, generally
  just to add "HTTP::" qualifiers.

- Now that the variables associated with FTP analysis are part of an
  "FTP" module, the "ftp_" prefix associated with:

	ftp_excessive_filename_len
	ftp_excessive_filename_trunc_len
	ftp_guest_ids
	ftp_hot_cmds
	ftp_hot_files
	ftp_hot_guest_files
	ftp_ignore_invalid_PORT
	ftp_ignore_privileged_PASVs
	ftp_log
	ftp_skip_hot

  has been removed, and these are now called:

	excessive_filename_len
	excessive_filename_trunc_len
	guest_ids
	hot_cmds
	hot_files
	hot_guest_files
	ignore_invalid_PORT
	ignore_privileged_PASVs
	log_file
	skip_hot

  To get to them from other scripts, you specify, for example,

	redef FTP::guest_ids = { .... };

  whereas before you had to use:

	redef FTP::ftp_guest_ids = { .... };

- The new connection logging format introduced in 0.8a57 is now the
  default, unless you redef the new variable "traditional_conn_format"
  to be T (Robin Sommer).  Connections using unidentified ephemeral
  ports now have a service of simply "other" rather than other-XXXX.
  The 'U' connection status flag has been removed (unless you're using
  traditional_conn_format).

- Tables can now be directly indexed by records, and indexing using records
  is no longer interchangeable with using a corresponding list of indices
  (Umesh Shankar).  This may require adjustments to existing policy
  scripts.

- Hostnames such as www.google.com now have type set[addr] rather than
  a funky internal list type.

- The new function dump_current_packet(file_name: string) dumps a copy of
  the current packet to the file with the given name, appending it if the
  file already exists (Robin Sommer).  The file is in tcpdump format.
  A handy use for this is in an event handler for signature_match(),
  to record packets that match given signatures.

- The event new_packet() is invoked for each new packet (Robin Sommer).
  It currently doesn't provide the packet contents but soon will in
  a fashion similar to secondary-filter.bro.

- "cf -f fmt" specifies a strtime() format.  -u specifics UTC time rather
  than local time (Mark Delow and Craig Leres).  cf now has a man page
  (Craig Leres).

- Two new variables, backdoor_ignore_local and backdoor_ignore_remote,
  can be used to specify backdoor signatures that should be ignored
  if the server is local/remote.

- A bug has been fixed in which a "next" executed in the final iteration
  of a for loop would mangle the subsequent processing of the outer
  statements (Chema Gonzalez).

- Bug fixes for MIME and Base64 processing (Ruoming Pang).

- pcap.bro now builds its filter in the opposite order (restrict_filters
  first), which can improve packet filtering performance (Robin Sommer).

- A bug in &default has been fixed.

- Portability for different pcap_compile_nopcap() calling sequences
  (Chema Gonzalez).

- Some tweaks for a minor reduction in memory consumption.

- A memory leak for secondary packet filters has been fixed.

- The localization of error messages (what script line they correspond to)
  has been improved.



More information about the Bro mailing list