[Bro] new Bro CURRENT release (0.9a8)

Vern Paxson vern at icir.org
Wed Feb 16 17:09:34 PST 2005


A new CURRENT release, 0.9a8, is now available from:

	ftp://ftp.ee.lbl.gov/bro-0.9-current.tar.gz

This release includes a large number of changes and bug fixes (appended).

		Vern


-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+


0.9a8

- aux/rst/ contains the source for the "rst" tool used by Bro (via the
  policy script function terminate_connection() in conn.bro) to tear
  down established connections by forging RST packets.

- Bro's main event loop has been reworked (Robin Sommer).  This should
  (1) not cause any visible differences in most cases, (2) improve
  performance in some cases, (3) fixed problems running Bro without
  a network input (but still receiving asynchronous input from remote
  event sources).  There are some more changes coming to this soon.

- Passive OS fingerprinting has been added, based on Michal Zalewski's
  "p0f" tool (Holger Dreger).  Currently, it's limited to fingerprinting
  clients based on the initial SYNs they send.  To use it, define
  an event handler:

	OS_version_found(c: connection, host: addr, OS: OS_version)

  OS_version is a record containing a string $genre (e.g., "Solaris"),
  a string $detail (e.g., "2.0.27"), a count $dist (hop-count distance
  from monitor to host), and $match_type, which specifies via an
  enumerated type whether the match was direct from a signature,
  generic to the genre, or "fuzzy".

  The match is done against a passive fingerprinting signature file,
  which is specified by the variable passive_fingerprint_file.
  It defaults to "sigs/p0fsyn", which is found using $BROPATH
  and has an "osf" suffix added.

  You can restrict the matching to only be performed for hosts from
  particular subnets by adding those subnets to the variable
  generate_OS_version_event.  If it's empty (default), then all subnets
  are analyzed.

  Note, the passive fingerprinting should be integrated with the
  version-tracking in software.bro, but this hasn't been done yet.

- Support for IPv6 has been repaired and brought up to date.  Note, however,
  that inter-Bro communication currently only works over IPv4.

- Signature-matching is now off by default in brolite.bro.  If you want
  to use it, define use_signatures = T prior to @load'ing it.

- Notices are now tied to their corresponding connections (Scott Campbell).

- New backdoor detectors for IRC, SMTP, Gaobot (Scott Campbell).

- Signature matches now have a connection associated with them (Scott Campbell).

- Bro scripts that set initial timers (via "schedule" statements in a
  bro_init handler) but don't have any source of network input (trace
  files or live interfaces) now execute in real-time, with network_time
  set to the current time, rather than having their timers expire immediately.

- Default timeouts have been added to tables in trw.bro and http.bro, which
  have been found operationally to potentially grow very large (Scott Campbell).

- The new policy script large-conns.bro can be included in order to
  track the size of TCP connections (each direction is referred to
  as a "flow") using a secondary packet filter (Chema Gonzalez).
  This method is completely separate from Bro's usual size accounting,
  and offers the advantages that it tracks sizes even for connections
  that don't terminate (or for which Bro misses their establishment)
  and for connections with sizes > 4 GB.

  The interface is via the function:

	function estimate_flow_size_and_remove(cid: conn_id, orig: bool)

  If $orig=T, then an estimate of the size of the forward (originator)
  direction is returned.  If $orig=F, then the reverse (responder) direction
  is returned.  In both cases, what's returned is a "flow_size_est" record,
  which includes a flag, $have_est, indicating whether there was any
  estimate formed. If $have_est is T, then the record also includes
  an estimate in bounded by $lower (lower bound) and $upper (upper bound).
  The estimate also includes $num_inconsistent, which, if > 0, means that
  the estimates came from sequence numbers that were inconsistent, and
  thus something is wrong - perhaps packet drops by the secondary filter).
  Finally, calling the function causes the flow's record to be deleted.

- An RSH analyzer has been contributed by Manu (ManuX at rstack.org).
  It generates rsh_request and rsh_reply events, and the following notices:

	  DifferentRSH_Usernames
		Client and server username differ.

	  FailedRSH_Authentication
		Attempt to authenticate via RSH failed.

	  InteractiveRSH
		The RSH session appears to be interactive (multiple
		lines of user commands).

	  SensitiveRSH_Input
	  SensitiveRSH_Output
		RSH client input or server output match input_trouble/
		full_input_trouble or output_trouble/full_output_trouble.

- The new notice action NOTICE_EMAIL indicates that in addition to
  logging an alarm, it should also be sent out as email (Scott Campbell).
  By default, email is only sent if Bro is running on live traffic;
  you can override this via redef'ing the script variable mail_notification.
  Mail is sent using the script specified by the mail_script variable
  (default: "mail_script.sh", which is now included in the distribution,
  but at present is not installed), which must be in $PATH.  The mail
  is sent to the username specified in mail_dest (default: the local
  "bro" user, though you can change this to name at domain).

  Note that specifying email as a separate notice action may change
  in the future, to instead be an attribute that's associated with
  other notice actions.  For example, it may make sense to want
  to specify both NOTICE_ALARM_PER_CONN and NOTICE_EMAIL; currently,
  however, you can't do this.

- A similar notice action NOTICE_PAGE does the same thing as NOTICE_EMAIL
  except it send the mail to mail_page_dest (Scott Campbell).

- You can now use the attribute &rotate_size for file objects to
  specify the maximum file size in bytes (Robin Sommer). If the limit
  is reached, the file is rotated similiarly as is already done with
  &rotate (which, for consistency, has been renamed to &rotate_interval).

  For both &rotate_size and &rotate_interval, when they trigger they
  now generate events (rotate_size and rotate_interval, respectively;
  each takes the file as the sole argument) rather than invoking
  &postprocessor, which has been removed.

  There's also a new variable log_rotate_size to set a global size maximum.

  Related to log rotation are the following new built-in functions:

	rotate_file(f: file) closes the file, moves it to a temporary
	name, and opens a new one. It returns the new "rotate_info"
	record, which gives the temporary name and the open/close times.

	rotate_file_by_name(s: string): similar, but call by the name
	of the file rather than a Bro script value.  This is needed
	because some files are not represented by file objects but need
	to be rotated nevertheless (most importantly, the tcpdump save
	file and the dump files for dump_current_packet()). This function
	rotates the file with the given name.

  Finally, you can load the new policy script rotate-logs.bro to
  get default behavior of rotating all log files every hour.

- The new "@unload <script>" directive specifies that future @load's of
  <script> should be skipped.  This is useful for overriding analyzers
  loaded by scripts that pull in a bunch of analysis.  For example,

	@unload ntp
	@load mt

  would load all of the "mt" analyzers *except* ntp.bro.

- The new built-in function get_file_name(f: file): string returns
  the filename associated with a file (John McNicholas).

- The new built-in function get_contents_file(id: conn_id, direction: count)
  returns the contents file (set using set_contents_file()) for the given
  direction (John McNicholas).

- The new built-ins time_to_double() and double_to_time() convert between
  double values and time values (Robin Sommer).  The new built-in floor()
  returns the floor of a double value; this returned value is also a double.
  Thus, floor(-3.4) returns -4.0.

- Support for sending packets between Bro's (Robin Sommer).

- Bro now has a geneal mechanism internal for traversing policy scripts
  (Umesh Shankar).  Various script analyses can be specified using the
  new -z flag.

  Currently, the one supported form of analysis is "-z notice", which
  prints all of the different types of notices that the script you've
  loaded can generate.  For example, "bro -z notice ftp" will generate:

	  Found NOTICE: BackscatterSeen
	  Found NOTICE: FTP_PrivPort
	  Found NOTICE: FTP_BadPort
	  Found NOTICE: PortScan
	  Found NOTICE: FTP_ExcessiveFilename
	  Found NOTICE: ScanSummary
	  Found NOTICE: AddressDropped
	  Found NOTICE: DroppedPackets
	  Found NOTICE: SensitiveConnection
	  Found NOTICE: FTP_UnexpectedConn
	  Found NOTICE: SSH_Overflow
	  Found NOTICE: FTP_Sensitive
	  Found NOTICE: TerminatingConnection
	  Found NOTICE: PasswordGuessing
	  Found NOTICE: AddressDropIgnored
	  Found NOTICE: AddressScan

- The signature rule language now supports an "active" keyword,
  which can be set to "true" or "false", with the latter turning
  off the rule (Roger Winslow).  If set to false the signature will
  not be loaded into the rule matcher, otherwise it is.

- The signature rule language now supports meta data of the form
  ".MMM<whitespace>XXX", where MMM is arbitrary text which makes up the name
  of the meta data option and where XXX is arbitrary text up to the end
  of the current line (Roger Winslow).  The intent is that some forms of
  meta-data will be regularized/standardized in the future - information
  such as date modifed, category/class, weighting, etc.  For now, it
  provides a way to annotate rules with nominally more structure than just
  using comments (though it is currently treated the same, i.e., everything
  is ignored).

- The following meta data option names are now reserved: .version, .revision,
  .version-date, .revision-date, .date-created, .location

- The new enumerated type "transport_proto" is used to specify different
  types of transport protocols: "tcp", "udp", "icmp", and "unknown_transport".
  Associated with it are new built-in functions: is_udp_prot(), is_icmp_port(),
  get_conn_transport_proto, and get_port_transport_proto.  The latter two
  map a given connection and a given "port" value to their corresponding
  transport_proto value.

- A bunch of tuning (regular expressions for sensitive login sessions,
  scan detection thresholds, forbidden/hot usernames and filenames,
  sensitive URIs, "weird" actions) have been incorporated from
  operational configurations.

- Serious bugs in managing large numbers of files fixed.

- A serious bug with negative DNS TTL settings (and, more generally,
  with negative timer values) fixed.

- The traditional connection logging format is no longer supported.

- The SMTP analyzer's state machine processing has been modified to
  correctly deal with clients that (incorrectly) pipeline their commands
  (Ruoming Pang).

- A bug fixed in detecting SMTP relays for connections w/o message bodies
  (Ruoming Pang).

- A bunch of bugs in recording connection summaries for UDP flows
  have been fixed.

- A new script module, passwords.bro, generates PasswordExposed notices
  for activity (currently just rlogin/telnet logins) that expose passwords.

- A new script module, file-flush.bro, can be loaded to cause all log
  files to be flushed every file_flush_interval (default: 10) seconds.
  This is handy if you like to watch the files in real time.

- Zone transfers now generate a ZoneTransfer notice, unless the host
  making the request is in DNS::zone_transfers_okay.

- Bro's DNS cache (generated using -P and accessed using -F) is
  now kept in the .state/ subdirectory rather than in the user's
  home directory (Roger Winslow).

- Some changes to remote propagation of events/values and detection of
  state inconsistencies (Robin Sommer).

- A fix for avoiding delays on low-volume links for some systems for
  which it can take a long time to fill up the pcap buffer, and pcap doesn't
  return partial buffers (Robin Sommer).

- A bug in table expiration timers has been fixed (Robin Sommer).

- A bug in comparing subnets has been fixed.

- A bug in using a non-constant value for a &write_expire attribute
  has been fixed.

- A bug in using CONTENTS_BOTH for writing reassembled streams to
  files has been fixed (John McNicholas).

- A subtle but potentially damaging bug in fragment reassembly has
  been fixed.

- A bug with using local variables of vector types has been fixed.

- A bug with comparing strings has been fixed.

- Bro no longer generates the RST_with_data "weird", as with modern
  stacks it's no longer any sort of strange occurrence.

- Related to this, the signature rule matcher no longer matches
  against the payload of RST packets.  (Note, this is an imcompatibility
  with Snort.)

- Portmapper mappings are now written in the connection log in
  alphabetical order.

- The event engine variable frag_timeout now defaults to 5 minutes if you
  load frag.bro, and is accessed via redef rather than by defining the
  global directly.

- The interval that signatures.bro waits for until generating a signature
  summary can now be set using the new script variable sig_summary_interval,
  and a bug in generating the summaries has been fixed (Robin Sommer).

- The new script peer-status.bro generates periodic "update" events regarding
  a remote peer's status (Robin Sommer).  These take the form:

	type peer_status: record {
		res: bro_resources;
		stats: net_stats;
		current_time: time;
		cpu: double;		# average CPU load since last update
		default_filter: string;	# default capture filter
	};

- The bro_resources record returned by resource_usage() now includes
  three additional fields, $version (the version of Bro), $debug
  (T if Bro was compiled with debugging information), and $start_time
  (the time Bro began executing - clock time, not network time).

- The new built-in function same_object(o1: any, o2: any): bool
  returns true if its arguments refer to the same object, false
  otherwise.  This can be useful for comparing tables, for example
  in calls to table element expiration functions.

- The new built-in function bro_is_terminating(): bool returns true if
  Bro is done reading from its network input source(s) and is now
  in its final termination cleanup (Robin Sommer).

- A new built-in strftime() formats a timestamp, returning a string
  (Robin Sommer).

- A new built-in file_size() returns the size of the file with a
  given name (Robin Sommer - note: *not* a Bro file value).

- A potential deadlock with inter-Bro communication has been fixed
  (Robin Sommer).

- Bro now always forks a copy of itself when executing, as this
  can save considerable memory when using inter-Bro communication
  (Robin Sommer).

- The Bro interconnection protocol now includes explicit handshaking
  during session establishment to mark that a peer is ready (Robin Sommer).
  Implementing this includes a change in the wire protocol that is
  incompatible with the protocol used in the past.

- The TCP inactivity timer is now started whenever a connection
  transitions from a pre-establishment state (including "inactive")
  to some sort of established state (Robin Sommer).  Prior to this
  fix, connections for which a proper SYN handshake was not seen would
  not be timed out as inactive.

- The --disable-openssl configure option has been removed; now
  the only option is --with-openssl, and --with-openssl=no disables
  use of OpenSSL (Gregor Maier).

- A bug in invoking &expire_func functions has been fixed (Robin Sommer).

- A bug in logfile rotation has been fixed (Robin Sommer).

- A bug in recognizing negative floating point values has been fixed.

- worm.bro now suppresses the default signature action for worms
  it knows about, since it generates events for them (Robin Sommer).
  The list of worms detected via signatures now includes Bagle-BC.

- Signatures for known worms are now skipped when doing signature
  summaries and scan detection, if worm.bro is loaded (Robin Sommer).

- request_remote_events and request_remote_sync now implicitly
  do set_accept_state, too.

- Better error handling for SSL connections (Robin Sommer).

- Bug fixed which caused diagnostic messages to be lost when using
  inter-Bro communication (Robin Sommer).

- gcc 3.4 portability fixes (Brian Lindauer).

- Solaris portability fixes (Robin Sommer).

- The Bro distribution now includes and uses its own version of libpcap
  for portability reasons (Jason Lee).

- Some minor bug fixes to handling of tcpdump save files (Robin Sommer).

- Detection added for a (now quite old) SSHv1 overflow attack.

- A bug in skipping processing of connections for large chunks of
  data has been fixd (Chema Gonzalez).

- Some memory leaks fixed (Robin Sommer).

- fmt()'s "%d" format now accepts values of enum types.



More information about the Bro mailing list