[Bro] new Bro CURRENT and STABLE releases (1.0 and 0.9a11)

Vern Paxson vern at icir.org
Sun Oct 23 20:13:20 PDT 2005


Bro release 1.0 is now available from:

	ftp://bro-ids.org/bro-1.x-current.tar.gz

This branch has become the new CURRENT release.  The 0.9 branch (formerly
CURRENT) is now the STABLE release, for which there is also a new release,
0.9a11, available from:

	ftp://bro-ids.org/bro-0.9-stable.tar.gz

The 1.0 release contains a significant number of new features, protocol
analyzers, and bug fixes, per the appended change log.  The 0.9a11 release
contains just a few bug fixes (at the bottom of this message); as a STABLE
branch, from now on 0.9 will only be updated with significant bug fixes.

The old STABLE release, based on the 0.8 branch, remains available at

	ftp://bro-ids.org/bro-pub-0.8-stable.tar.gz

We do not anticipate making any further changes to it.

		Vern


-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+


1.0 Sun Oct 23 17:27:45 PDT 2005

- Bro now includes BinPAC (Binary Protocol Analyzer Compiler), a language
  and compiler for automating the construction of analyzers for binary
  protocols (Ruoming Pang).

- Ruoming has used BinPAC to rewrite the analyzers for DCE/RPC (with
  significant enhancements, including adding the endpoing mapper) and SMB
  (likewise enhanced and bug-fixed), and creating a new analyzer for NCP
  (Netware Core Protocol).  The NCP analyzer generates two events:

	ncp_request(c: connection, frame_type: count, length: count,
			func: count)

	ncp_reply(c: connection, frame_type: count, length: count,
			req_frame: count, req_func: count,
			completion_code: count)

- The beginnings of an analyzer for NetBIOS name service (Ruoming Pang).
  It generates the following events:

	event nbns_standard_name_query(c: connection)
	event nbns_nbstat_query(c: connection)
	event nbns_name_reg_request(c: connection)
	event nbns_nb_name_query(c: connection)

- New IRC analyzer (Roland Gruber).  It generates a lot of events; see
  policy/irc.bro.  Note, the formatting of the log file will at some point
  be changed to be more uniform and streamlined.

- ICMP events now include an initial parameter of type "connection",
  the same as for TCP & UDP flows (Ruoming Pang).  This facilitates
  traffic analysis by associating generic connection events such as
  connection_state_remove with ICMP events.  This affects:

	event icmp_sent(c: connection, icmp: icmp_conn)
	event icmp_echo_request(c: connection, icmp: icmp_conn,
			id: count, seq: count, payload: string)
	event icmp_echo_reply(c: connection, icmp: icmp_conn, id: count,
			seq: count, payload: string)
	event icmp_unreachable(c: connection, icmp: icmp_conn,
			code: count, context: icmp_context)
	event icmp_time_exceeded(c: connection, icmp: icmp_conn,
			code: count, context: icmp_context)

- New POP3 analyzer (Florian Schimandl, Hugh Dollman and Robin Sommer).
  Loading pop3.bro analyzes the protocol messages, and loading mime-pop.bro
  also extracts the email headers and content.

- New events (Ruoming Pang):

	connection_first_ACK(c: connection)
		generated upon the ACK completing a TCP handshake. Useful
		in detecting "blink scans" (a FIN coming from the client
		right after the ACK)

	tcp_rexmit(c: connection, is_orig: bool, seq: count, len: count,
			data_in_flight: count, window: count)
		generated when a TCP sender retransmits data

	rpc_call(c: connection, prog: count, ver: count, proc: count,
			status: count, start_time: time,
			call_len: count, reply_len: count)
		can be used to process RPC calls in a generic fashion

	nfs_reply_status(n: connection, status: count)
		supplies the status of NFS server replies

	netbios_session_raw_message(c: connection, is_orig: bool, msg: string)
		access to a NetBIOS SSN message in raw terms

	smb_get_dfs_referral(c: connection, max_referral_level: count,
				file_name: string)
		generated for SMB DFS referal requests
 
	dns_message(c: connection, is_orig: bool, msg: dns_msg, len: count)
		low-level event generated for each DNS request/reponse

	dce_rpc_bind(c: connection, uuid: string)
		generated for DCE RPC binds

	dce_rpc_message(c: connection, is_orig: bool, ptype: dce_rpc_ptype,
			msg: string)
		low-level access to DCE RPC messages; see const.bif for
		dce_rpc_ptype values

	epm_map_response(c: connection, uuid: string, p: port, h: addr)
		reply from DCE portmapper

- New operator |x|, a sizeof operator (Christian Kreibich).  It yields
  absolute values for numerical values, file size for files, the number
  of enums for an enum type, the number of addresses in subnets, number
  of fields in records, numeric equivalent for addresses, the number of
  elements in vectors/sets/tables, and the length of strings.

- A new clone operator, "* <expr>", produces deep copies of aggregate
  values and the usual duplicates for atomic ones (Christian Kreibich).
  For example, while:

	   1	type foo: record {
	   2		c: count;
	   3		s: string;
	   4	};
	   5
	   6	f1$c = 10;
	   7	f1$s = "Hello";
	   8
	   9	f2 = f1;
	   10	f1$c = 20;
	   11	f1$s = "World";
	   12
	   13	print fmt("%d/%s %d/%s", f1$c, f1$s, f2$c, f2$s);

  yields "20/World 20/World", changing line 9 to:

	f2 = *f1;

  yields "20/World 10/Hello".

- New operators "+=" and "-=", which work on both numerical values and
  strings (Christian Kreibich).

- "+" now works on strings: s1 + s2 yields the concatenation of both
  (Christian Kreibich).

- You can now express the equivalent of ICMP "port numbers" using
  <number>/icmp, where <number> is the ICMP type (Ruoming Pang).

- Bro now accepts long versions of options, such as --readfile for -r
  (Christian Kreibich).

- Bro now has a "pseudo-realtime" mode, activated by --pseudo-realtime,
  that causes it to mimic real-time operation when executing against
  a trace (Robin Sommer).  This is useful for evaluating performance in
  a controlled fashion.

- SMTP analyzer changes (Roger Winslow): support for 554 code in RCPT
  responses; logging when the server refuses the argument to RCPT;
  support for 502 code in response to a HELP command.

- Addition of two universal hash functions: H3 (from David Moore, based
  on code by Ken Keys) and TwoWise (Dietzfelbinger, from Yin Zhang)
  (Ruoming Pang).  Use --enable-h3 and --enable-dietzfelbinger to enable
  them (used as the hash function for short data).  H3 is used by default.

- The "bif" compiler for compiling Bro built-in functions now supports
  an "enum" type (Ruoming Pang).  The syntax is:

	enum dce_rpc_ptype 
	%{
		DCE_RPC_REQUEST,
		DCE_RPC_PING,
		DCE_RPC_RESPONSE,
	%}

  which is translated to an enum declaration of "dce_rpc_ptype" in
  Bro, an EnumType* enum_dce_rpc_ptype in NetVar.{h,cc} and a C++ enum
  BroEnum::dce_rpc_ptype {...}. 

  One limitation is that redef's on enum types cannot be taken into
  account because the bif is parsed at compile time.

- 64-bit integer support via --enable-int64 (Ruoming Pang).

- The new, experimental policy script adu.bro provides a generic way to
  extract application-layer ADUs (Christian Kreibich).  It heuristicly
  groups blocks of content sent from one side to another, uninterrupted
  by any data in the opposite direction, into an approximate ADU (request
  or reply).  These then generate adu_tx (originator -> responder) and
  adu_rx (responder -> originator) events.  You can control on which ports
  it does this analysis, as well as the amount of data inspected nad
  grouped, using variables documented in the script.

- The new built-in function

	function str_smith_waterman(s1: string, s2: string, params: sw_params)
	: sw_substring_vec

  computes the Smith-Waterman overlap between two strings (Christian Kreibich).
  The third parameter is a record with two fields, $min_toklen (minimum
  length for common tokens) and $sw_variant, which takes a value of 0
  for single-matching and 1 for multiple-matching.

  The return value is a vector of sw_substring records, which hold the
  following fields:

	str: string;	# the common subsequence
	index1: count;	# where it occurs in input string 1
	index2: count;	# where it occurs in input string 2
	new: bool;	# true if start of new alignment

- If you set the new control variable record_state_history to T, then
  connections recorded to the conn.$BRO_LOG_SUFFIX log file will include
  a field that shows the different states encountered during the connection
  (Mark Allman):

	Symbol	State
	------	-----
	S	Initial SYN seen for TCP connection.
	H	SYN-ACK seen for TCP connection.
	D	Data packet seen (TCP or UDP).
	A	Pure ACK seen for TCP connection.
	F	FIN seen for TCP connection.
	R	RST seen for TCP connection.
	I	TCP connection included a FIN+RST packet.
	Q	TCP connection included a packet with multiple connection
		control flags other than FIN+RST (e.g., SYN+RST).
	C	Connection included one or more packets with failed checksums
		(TCP or UDP).

  The symbols are printed in upper-case for connection originators and
  lower-case for responders.  The S/H/F/R symbols are also repeated if
  Bro sees the corresponding control packet subsequently with a different
  sequence number.

  For example, a TCP connection which Bro saw from the beginning, i.e.,
  a normal establishment, followed by the client (originator) first sending
  data, then the server responding, followed by the server initiating a
  normal close which the client then completes, will be annotated as
  "ShADadfF".

- The "for" looping construct now can be used to iterate over the non-empty
  indices of a vector (Christian Kreibich).

- If you set the new variable skip_http_data to T (default: F), then the
  HTTP analyzer will attempt to not reassemble the data portions of HTTP
  request/responses (Ruoming Pang).  This can be a performance benefit
  in environments with high volumes of HTTP traffic, though it may not be
  a large win if the processing is dominated by executing the policy script.

- The new built-in

	remask_addr(a1: addr, a2: addr, top_bits_from_a1: count): addr

  take some top bits (e.g. subnet address) from a1 and remaining bits
  (intra-subnet part) from a2 and merge them to get a new address (Ruoming
  Pang).  Useful for anonymizing at the subnet level while preserving
  serial scans.

- The new built-in

	decode_netbios_name(name: string): string

  takes a string in NetBIOS encoding and returns its original form
  (Ruoming Pang).

- The new variable ignore_keep_alive_rexmit controls whether to
  include keep-alives when counting retransmitted packets (Ruoming Pang).
  It defaults to F (i.e., do count them).

- The calling sequence of dce_rpc_request and dce_rpc_reply have changed to:

	event dce_rpc_request(c: connection, opnum: count, stub: string)
	event dce_rpc_response(c: connection, opnum: count, stub: string)

  (Ruoming Pang).  Use dce_rpc_message to get access to the RPC type and
  the raw message.

- The calling sequence of the netbios_session_message event has changed to:

	netbios_ssn_message(c: connection, is_orig: bool,
				msg_type: count, data_len: count)

  (Ruoming Pang).  Previously it was parameterized with the connection
  and the raw message (now available via netbios_session_raw_message).

- The calling sequences of smb_com_{read,write}_andx have changed to
  no longer include the is_orig parameter because it is in fact fixed for
  these events (Ruoming Pang).

- The calling sequence of smb_message has changed (Ruoming Pang) to:

	smb_message(c: connection, is_orig: bool, cmd: string,
			body_length: count)

- Bug fix specifying the &default value for tables that yield function
  values (Ruoming Pang).  For example:

	type tcp_content_handler_func:
		function (c: connection, is_orig: bool, seq: count,
				contents: string);

	function default_tcp_content_handler(c: connection, is_orig: bool,
						seq: count, contents: string)
		{
		# do something ...
		}

	const tcp_content_orig_handlers: table[port] of
		tcp_content_handler_func = {} &redef &default =
			default_tcp_content_handler;

  Previously, Bro would take the function given with &default as the default
  function to call when accessing a missing element, rather than a default
  *value* to directly return. Bro now checks the value type against the
  function type to see if they match in type.

- The new variables forward_remote_events and forward_remote_state_changes
  specify whether to broadcast events/state received from one peer to other
  peers (Robin Sommer).  Both default to F.  Note, these options are temporary;
  they will disappear when we add a more sophisticated script-level
  communication framework. 

- Vectors can now be initialized using the syntax such as

	global foo: vector of string = ["foo","bar"];

  (Robin Sommer).

- Bug fixes for &synchronize'ing vectors (Robin Sommer).

- The internal implementation of strings in the policy language has
  been heavily revamped (Christian Kreibich).

- String built-in functions are now in strings.bif rather than bro.bif
  (Christian Kreibich).  This includes two new built-ins:

	str_split(s: string, idx: index_vec): string_vec
	strstr(big: string, little: string): count

  string_vec is a new policy script type that is an alias for
  "vector of string".

- The new options --load-seeds <file> and --save-seeds <file> let you
  record Bro's seeds to a file and then re-use these seeds in a later
  invocation (Christian Kreibich). The primary intended usage is to
  provide determinism in hash table iterations etc. for debugging purposes.

- Communication protocol changes (Robin Sommer):

  * Internal PING/PONG messages to measure round-trip times.  The new script
    remote-ping.bro issues PINGs every second and logs to remote.log.

  * Optional data compression if libz is available.  Remote::Peer$compression
    specifies compression level, with no compression being the default.

  * Inter-Bro communication is now performed in four explicit phases:

    //  Setup:
    //      Initial phase.
    //      VERSION messages must be exchanged.
    //      Ends when both peers have sent VERSION.
    //  Handshake:
    //      REQUEST_EVENTS/REQUEST_SYNC/CAPTURE_FILTER/CAPS/selected SERIALs 
    //        may be exchanged.
    //      Phase ends when both peers have sent PHASE_DONE.
    //  State synchronization:
    //      Entered iff at least one of the peers has sent REQUEST_SYNC.    
    //      The peer with the smallest runtime (incl. in VERSION msg) sends
    //        SERIAL messages comprising all of its state.
    //      Phase ends when peer sends another PHASE_DONE.
    //  Running:
    //      Peers exchange SERIAL (and PING/PONG) messages.
    //      Phase ends with connection tear-down by one of the peers.

  * Serializing network packets includes textual tags for identification.

  * Serializing files includes the state of buffering.

- Pending events for remote peers are now flushed when Bro terminates,
  and the net_done event is *not* propagated to peers (Robin Sommer).

- Makefile.am cleanups (Christian Kreibich).

- libpcap portability fix for OpenBSD (Gordon Willem Klok).

- Performance bug fix for SMTP relay detection (Vern Paxson).

- sprintf -> snprintf tweak (Vern Paxson).

- Bug fix for serializer regular-expression matchers (Robin Sommer).

- Some fixes for access to uninitialized variables/state (Christian Kreibich
  and Vern Paxson).

- More informative messages for some internal errors (Christian Kreibich).

- Bug fixes for implementation of vectors (Christian Kreibich).

- Fixes for FreeBSD 5 installs (Jason Lee).

- gcc 4.0 compatibility (Christian Kreibich).

- Bug fix for correctly propagating libpcap failures (Chema Gonzalez).

- Bug fixes for prefix-preserving IP address anonymization (Chema Gonzalez).

- The MIME analyzer in mime.bro is now in "module MIME" (Vern Paxson).

- Bug fix for the IRC backdoor detector (Scott Campbell).

- The capture filter used for NFS traffic now includes UDP fragments,
  since NFS UDP traffic is often fragmented (Ruoming Pang).

- New internal mechanisms to suspend/resume processing to enable a Bro
  receiving synchronized state to put its own packet processing on hold
  (Robin Sommer).

- A bug with the serialization cache not being used for modified objects
  has been fixed (Robin Sommer).

- A number of enhancements to inter-Bro communication performance and error
  handling improved (Robin Sommer).

- Internal restructuring to fix problems with dispatching packets when
  using the packet sorter (Ruoming Pang).

- Christian Kreibich has contributed a number of fixes for code flaws
  such as potentially unsafe library calls.


-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+


0.9a11 Sun Oct 23 18:20:31 PDT 2005

- libpcap portability fix for OpenBSD (Gordon Willem Klok).

- Performance fix for high-volume SMTP relay detection (Vern Paxson).

- Fix for bro.rc script configuration (Roger Winslow).

- Fix for IRC backdoor detector (Scott Campbell).



More information about the Bro mailing list