[Bro] new Bro CURRENT and STABLE releases (1.2 and 1.1)

[Vern Paxson]

Bro release 1.2 is now available from:

	ftp://bro-ids.org/bro-1.x-current.tar.gz

This becomes the new CURRENT release.  The 1.1 branch (formerly CURRENT)
is now the STABLE release:

	ftp://bro-ids.org/bro-1.1-stable.tar.gz

The most significant new features with 1.2 are dynamic protocol detection
and a large set of enhancements to the BinPAC system for generating protocol
analyzers.  The appended changelog lists numerous other features/changes/fixes.

The old STABLE release, based on the 0.9 release, remains available at

	ftp://bro-ids.org/bro-pub-0.9-stable.tar.gz

We do not anticipate making any further changes to it.

		Vern


-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+


1.2 Tue Oct 17 12:09:49 PDT 2006

- Bro now supports DPD, dynamic protocol detection (Robin Sommer, Holger
  Dreger, and Michael Mai).  With DPD, Bro can analyze protocols regardless
  of what port numbers they use: it infers the protocol based on which
  application analyzers can parse it without error.  Adding this functionality
  involved extensive changes to Bro's internals, but also now enables
  multiple Bro analyzers to work on the same connection, either concurrently
  or one nested inside the other (we have not taken much advantage of this
  latter capability yet, but see the FTP events discussed below).

  There are a number of new policy scripts, events, and variables associated
  with DPD processing, as follows.

  Scripts:

	You activate DPD by @load'ing dpd.bro.  It in turn instructs Bro
	to load the signature file policy/sigs/dpd.sig.  Note that Bro
	uses signatures to expedite deciding which analyzers to try on
	a given connection; it does *not* simply use the signatures to
	make the determination of which protocol is in use, as this is
	insufficiently robust.  (At this point, Bro provides signatures
	for FTP, IRC, HTTP, SMTP, and SSH.  In the future we plan to add
	other protocols.)

	Along with dpd.bro, you need to @load detect-protocols.bro or
	detect-protocols-http.bro. The former enables general detection
	of application-layer protocols, while the latter does further
	inspection of HTTP sessions to characterize applications running
	on top of HTTP such as Gnutella or SOAP.   (Loading dpd.bro
	is separate from loading one of these scripts because in principle
	Bro could use a different means than signatures to activate
	the analyzers, although currently it does not.)

	If you @load dyn-disable.bro, then once an analyzer determines
	that it does not match a given connection, it is deactivated
	(and a Notice is generated).  Otherwise, it still proceeds to try
	its best to analyze the connection (to possibly be more robust
	against evasion).

	The scripts dce.bro and smb.bro enable DPD for the Windows DCE and
	SMB protocols, respectively.  (Note that analysis of these protocols
	is undergoing a major expansion, not yet complete.)

  Events:

	event protocol_confirmation(c: connection, atype: count, aid: count)
		Generated when the given connection has been confirmed as
		conforming with the application type (protocol) specified
		by atype. aid is a globally unique analyzer ID that identifies
		a particular analyzer instance.

		The values for atype are symbolic names associated with
		each of Bro's analyzers, such as ANALYZER_IRC.  See the
		initialization at the beginning of Analyzer.cc for the
		full set of names.

		The function analyzer_name(atype: count): string translates
		these symbolic names into text.  For example,

			analyzer_name(ANALYZER_IRC)

		yields "IRC".

	event protocol_violation(c: connection, atype: count, aid: count,
				reason: string)
		Generated when the given connection has been found to
		violate the protocol of the given application type, with
		"reason" giving details.

  Variables:

	dpd_buffer_size: count (default 1024)
		Specifies how much pending data Bro keeps for connections
		that have not been classified yet.  Once this fills, the
		data is deleted, though classification can still continue
		(see below).

	dpd_match_only_beginning: bool (default T)
		If set, specifies that Bro should stop signature matching
		if it has processed dpd_buffer_size bytes.

	dpd_ignore_ports: bool (default F)
		If set, then Bro does not take into consideration the port
		numbers associated with connections when attempting to
		classify them (which can otherwise help the process in
		some cases).

	dpd_reassemble_first_packets: bool (default T)
		If set, then Bro does TCP stream reassembly before applying
		signature-matching to detect protocols.

	likely_server_ports: set[port]
		Specifies a list of ports that Bro will consider as likely
		used by servers.  For example, if Bro sees a connection
		that has already been established (so it does not know
		which side sent the initial SYN), and one side uses a port
		in this set, then it will assume that that side is the
		server (connection responder).  The set is empty unless
		you populate it or @load server-ports.bro, which specifies
		a large number of values.

	dpd_config: table[AnalyzerTag] of dpd_protocol_config
		Specifies the DPD configuration associated with each tag.
		The type dpd_protocol_config is simply:

			type dpd_protocol_config: record {
				 ports: set[port] &optional;
			};

		i.e., an optional $ports field specifying a set of ports
		associatd with the tag.  For example, ftp.bro now includes
		the equivalent of:

			redef dpd_config += {
				[ANALYZER_FTP] = [$ports = 21/tcp]
			};

  Functions:

	The function

		expect_connection(orig: addr, resp: addr, resp_p: port,
					analyzer: count, tout: interval)

	is called to alert Bro that a new connection is expected, initiated
	by orig to a server running on resp's port resp_p (note: orig's port
	is not specified) which will correspond to the specified analyzer
	(e.g., "FILE", which is used to analyze files transferred by FTP -
	see next item).  "tout" is a timeout to associate with the waiting.

	The function

		function disable_analyzer(cid: conn_id, aid: count)

	instructs Bro to disable the analyzer that generated the current
	event, assuming the analyzer is associated with the given connection
	ID.  This is used by the dyn-disable.bro script discussed above.

- A much more complete BinPAC compiler, along with new HTTP, DNS, and
  RPC/Portmap analyzers in binpac (Ruoming Pang). The flag "--use-binpac"
  activates the BinPAC-based analyzers (currently for HTTP and DNS).
  See www.cs.princeton.edu/~rpang/binpac-paper.pdf for a description of
  BinPAC, and let Ruoming know if you are interested in using BinPAC to build
  new analyzers.

- A new type of analyzer, FILE, analyzes the contents of a connection as
  though it were a data file (Robin Sommer).  Currently, it can generate
  two events:

	event file_transferred(c: connection, prefix: string, descr: string,
				mime_type: string)
		Indicates that the connection transferred a file. "prefix"
		is the beginning of the file's data; "descr" and "mime_type"
		are indicators of the file's type, as reported by the
		"libmagic" library.

		descr/mime_type are only set if Bro is configured on a
		system that includes the "libmagic" library.

	event file_virus(c: connection, virname: string)
		Indicates the connection transferred an executable
		corresponding to a known virus of the given name.

		This functionality is only available if Bro is configured
		on a system that includes the "libclamav" library.

  Note, this analyzer is enabled via a call to expect_connection by
  the FTP analyzer.

- New events relating to IRC analysis (Robin Sommer):

	event irc_client(c: connection, prefix: string, data: string)
		Generated upon seing a client message sent over the given
		IRC connection.  "prefix" is the command's prefix as defined
		by the IRC protocol.  It is used by servers to indicate the
		true origin of the message; it may be empty.  "data" contains
		the message.

	event irc_server(c: connection, prefix: string, data: string)
		Same for server messages.

	event irc_user_message(c: connection, user: string, host: string,
				server: string, real_name: string)
		Generated upon seeing an IRC "USER" command.

	event irc_password_message(c: connection, password: string)
		Generated upon seeing an IRC "PASS" command.

	event irc_channel_topic(c: connection, channel: string, topic: string)
		Generated upon seeing an IRC server reply that includes
		the channel topic.

	event irc_global_users(c: connection, prefix: string, msg: string)
		Generated upon seeing an IRC server reply that includes
		a count of the number of IRC users.

- The new experimental script irc-bot.bro tracks IRC-based bots (Robin Sommer).
  The accompanying script irc-bot-syslog.bro syslog's the state of the
  bot analysis every IrcBot::summary_interval seconds (default 1 minute).

- The new script proxy.bro looks for open Web proxies by matching incoming
  requests to a server with outgoing requests it makes (Robin Sommer).  It
  generates HTTPProxyFound Notices when it finds one.

- Changes to notices.bro (Robin Sommer):

	- notice_policy_item's now have a default $result of
	  NOTICE_FILE and a default $priority of 1.

	- The new notice_action_filter, notice_alarm_per_orig, alarms
	  on the first NoticeType from a specific source.  Subsequent
	  instances are tallied.

	- notice_action_filters now reside in the new script
	  notice-action-filter.bro (automatically loaded by notice.bro).
        
	- The notice actions NOTICE_ALARM_PER_CONN, NOTICE_ALARM_PER_ORIG,
	  and NOTICE_ALARM_ONCE have been removed, as they were never
	  actually implemented.

	- If the notice_policy returns IGNORE or FILE, the action_filters
	  filters are no longer consulted.

- A new attribute for tables and sets, &mergeable, changes the semantics
  of assignments, as follows (Robin Sommer).  Given two &mergeable tables/sets
  A and B, an assignment "A = B" becomes actually a join "A = A \cup B"
  (i.e., union).  The envisoned use is to help avoid race conditions
  when doing remote state synchronization.

- The semantics of &synchronized expire_funcs has changed (Robin Sommer).
  Now, when a table entry is expired and the operation is propagated to a
  a peer, the peer will call its expire_function.

- TRW analysis now skips UDP traffic because it currently treats
  all UDP connections as failures (Robin Sommer).
     
- trw.bro has been split into trw-impl.bro (the algorithm) and
  trw.bro (which simply activates the analysis), to facilitate writing
  scripts that have hooks into TRW analysis but don't presume it's
  active (Robin Sommer).

- The option report_remote_notices in remote.bro has been replaced
  by a new script you include, remote-report-notices.bro (Robin Sommer).

- The new function connect_peer() explicitly connects to a remote host
  (Robin Sommer).

- The new script remote-send-id.bro sends the current value of an ID
  to a remote Bro and then terminates processing (Robin Sommer).  It's
  intended for use from the command-line, as in

	bro -e "redef dst="<dst>" id="<name-of-id>" <scripts> remote-send-id

  The other scripts must set up the connection. <dst> is an index into
  Remote::destinations corresponding to the destination.

- New built-ins {suspend,resume}_state_updates() can be called to
  temporarily avoid propagating updates to &sync'ed values (Robin Sommer).
  This can avoid duplicated activity.

- The new function terminate_communication() instructs Bro to end its
  communication with remote peers (Robin Sommer).

- The new event remote_state_access_performed is raised when remote state
  access has been performed (Robin Sommer).  This is primarily for debugging.

- The log() built-in has been renamed to ln() to avoid conflict (Vern Paxson).

- bifcl now generates event generation wrapper functions from event.bif
  (Ruoming Pang).  For example, to generate event http_reply, currently
  one writes:

		val_list* vl = new val_list;
		vl->append(BuildConnVal());
		vl->append(new StringVal(fmt("%.1f", reply_version)));
		vl->append(new Val(reply_code, TYPE_COUNT));
		if ( reply_reason_phrase )
			vl->append(reply_reason_phrase);
		else
			vl->append(new StringVal("<empty>"));
		ConnectionEvent(http_reply, vl);

  In the future, one will be able to just call bro_event_http_reply(), and
  the code generated by bifcl looks like:

	void bro_event_http_reply(Connection* c, StringVal* version,
					bro_uint_t code, StringVal* reason)
		{
		val_list* vl = new val_list;

		vl->append(c->BuildConnVal());
		vl->append(version);
		vl->append(new Val(code, TYPE_COUNT));
		vl->append(reason);

		mgr.QueueEvent(http_reply, vl, SOURCE_LOCAL, c);
		}

  Accompanying this change is a semantic shift to types "string" and "port"
  in .bif files.  They used to be translated to C++ types BroString* and
  uint32, respectively.  Now they are translated to StringVal* and PortVal*.
  The functions in bro.bif are changed accordingly, and please be aware
  of this change when you write built-in functions in future.

  Also for this change, the parameter 'new' for rsh_request has been renamed
  'new_session', as 'new' is a reserved word for C++.

- Some ICMP "connections" now have services identified ("icmp-echo",
  "icmp-unreach") rather than just listing the service as "other"
  (Ruoming Pang).

- The new option remote_trace_sync_interval specifies an interval after
  which each Bro will stop processing its trace and wait for all others
  to signal that they have reached the same time (Robin Sommer).  The
  intent is support for operating Bro in a distributed cluster fashion
  (and in particular for debugging such clusters when running off-line
  on traces).

  This option only works in pseudo-realtime mode, and requires the new
  global remote_trace_sync_peers to give the total number of remote peers
  (not including self).  Signaling is done via a new communication message
  type.

- Extensions for DNS transformation/anonymization, including introduction
  of trace transformation for protocols other than TCP (Jason Lee).
  Not yet fully developed/debugged.

- Extensions for HTTP transformation/anonymization (Martin Casado).
  Not yet fully developed/debugged.

- The $conn field is now included in HTTPProxyFound notices (Robin Sommer).

- Changed service inference algorithm to favor lower-numbered
  likely-servers over higher-numbered ones (Vern Paxson).

- In pseudo-realtime mode, Bro now uses real-time for deciding which
  peer should send state (Robin Sommer).

- Time synchronization for Bro's running on traces in pseudo-realtime mode
  added (Robin Sommer).

- Avoidance of false content gaps improved when sorting packets with
  out-of-order timestamps (Ruoming Pang).

- Packets from the packet sorter are now more robustly drained upon
  termination of input (Ruoming Pang).

- Documentation for deep-copy updated (Christian Kreibich).

- Nasty fragment reassembly bug fixed (Vern Paxson).

- Serious bugs in EDNS0 processing fixed (Vern Paxson).

- Fixed significant misfeature of interconn.bro that stopped all processing
  of a connection once it makes a detection (Vern Paxson).

- Fixes for &read_expire operation across synchronizes tables (Robin Sommer).

- Fixes for multiple peers exchanging initial &sync state simultaneously
  (Robin Sommer).

- Improvements to graceful termination of Bro when communicating with
  remote peers (Robin Sommer).

- Fix for ICMP analyzer not always generating icmp_sent events
  (Robin Sommer).  This appears to still need some work, as now
  it generates redundant events.

- Fix for initial exchange of &sync state which could lead to 
  referencing unknown IDs (Robin Sommer).

- Fix to scan detection for differing semantics of connection compressor
  vs. non-compressor (Robin Sommer).

- Bug fix for distinguishing regular expression matches of length 0 from
  those of length 1 (Ruoming Pang).

- Fix for SSH version parsing in the presence of content gaps (Robin Sommer).

- Bug fix for IRC that could lead to crashes (Robin Sommer).

- Bug fix to refrain from adding new timers when a connection has
  already been removed from the connection table (Robin Sommer).

- Bug fix for packet_contents not including the transport-layer header
  (Robin Sommer).

- Some memory leaks fixed (Robin Sommer).

- A bunch of portability and distribution problems fixed (Christian
  Kreibich, Robin Sommer, Vern Paxson).