new bro "CURRENT" release - 0.8a79

[Vern Paxson]

An updated "CURRENT" version of Bro is now available from the usual location:

	ftp://ftp.ee.lbl.gov/bro-pub-0.8-current.tar.gz

This version has a lot of changes, including new analyzers, documentation,
language features, VLAN support, and the beginnings of IDMEF support.  I've
appended the changes since the last "CURRENT" version (0.8a70).

                Vern


-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+


0.8a79 Wed Mar 24 22:02:53 PST 2004

- Bro now has an SSL analyzer, written by Michael Kuhn and Benedikt Ostermaier,
  with further development by Scott Campbell.  It generates the following
  events:

	event process_X509_extensions(c: connection, ex: X509_extension)
	event ssl_X509_error(c: connection, err: int, err_string: string)
	event ssl_certificate(c: connection, cert: X509, is_server: bool)
	event ssl_certificate_seen(c: connection, is_server: bool)
	event ssl_conn_alert(c: connection, version: count, level: count,
				description: count)
	event ssl_conn_attempt(c: connection, version: int)
	event ssl_conn_established(c: connection, version: int,
				cipher_suite: count)
	event ssl_conn_reused(c: connection, session_id: sessionID)
	event ssl_conn_server_reply(c: connection, version: int)
	event ssl_conn_weak(name: string, c: connection)
	event ssl_session_insertion(c: connection, id: sessionID)

  Note, it still has a lot of rough edges; particularly, handling
  non-conformant input.  It also generates unnecessary ContentGap alerts
  due to the way it runs multiple analyzers (SSLv2 and SSLv3) on a single
  connection.  This will be fixed in the fairly near-term future.

- The manual has been updated with chapters on signatures (Robin Sommer)
  and using the interactive debugger (Umesh Shankar), along with a partial
  description of the new SSL analyzer (Michael Kuhn and Benedikt Ostermaier)
  and a number of updates to the documentation of built-in functions (Umesh
  Shankar), though this latter is still not complete since Umesh actually
  contributed this quite a while ago.

- Ruoming Pang has contributed a crude analyzer for DCE/RPC (used for Windows).
  It generates simple dce_rpc_request and dce_rpc_reply events.  It should
  not be considered stable.

- The traditional connection logging format (traditional_conn_format)
  is no longer the default.  The 0.8a70 release notes stated that this
  was the case but this time it really is :-).

- An experimental "vector" type has been added (Umesh Shankar).  A vector
  is an aggregate type.  For example:

	local suspect_hosts: vector of addr;

  You can randomly access elements starting with the first as 1, e.g.,

	suspect_hosts[1] = 10.0.0.8;

  and can also add elements at later postions even if there are gaps:

	suspect_hosts[31337] = 10.0.0.9;

  *The semantics and capabilities of vectors will be changing considerably.*

- Umesh Shankar has developed a framework for generating IDMEF messages.
  Currently it needs a modified version of libidmef, which is not included
  in this distribution.  Contact me or Umesh if you want a copy.

- A new attribute &synchronized causes the given global variable to
  be *synchronized* across concurrent instances of Bro (which are
  intercommunicating via remote.bro).  Any change made by one of them
  to the variable will be reflected (soon after) in the copy at
  the others.  A new event remote_state_inconsistency is generated
  if two instances both change the value before they're synchronized.
  (Contributed by Robin Sommer.)

- trw.bro implements a new scan detection algorithm, Threshold Random Walk
  (Jaeyeon Jung).  It's described in an upcoming IEEE S&P symposium paper.
  The analyzer generates two events:

	TRWAddressScan, # source flagged as scanner by TRW algorithm
	TRWScanSummary, # summary of scanning activities reported by TRW

  TRW is generally much more sensitive than Bro's regular detection algorithm.

- vlan.bro provides support for VLAN encapsulation.  More generally, Bro
  now has support for any sort of constant-offset encapsulation (Vinod
  Yegneswaran).  You specify the header size by redef'ing encap_hdr_size.
  You can also redef tunnel_port to be a UDP port which Bro treats as being
  the encapsulation (in the packet stream - not addressed to it) rather
  than all traffic.

- If you turn on statistics (i.e., via @load statistics) and also redef
  segment_profiling to T, then Bro will generate to the statistics file
  a trace of its "segment" processing.  A segment is a unit of internal
  execution.  Profiles look like:

	1058517499.615430 segment-processing-packet dt=0.000013 dmem=0
	1058517499.615430 segment-draining-events dt=0.000012 dmem=0
	1058517499.615671 segment-expiring-timers dt=0.000010 dmem=0
	1058517499.615671 segment-processing-packet dt=0.000010 dmem=0
	1058517499.615671 segment-draining-events dt=0.000012 dmem=0
	1058517499.615671 segment-policy/conn.bro:282 dt=0.000011 dmem=0
	1058517499.615671 segment-policy/conn.bro:253 dt=0.000012 dmem=0

  The first line states that at the given (packet) timestamp, the event
  engine processed a packet, taking 13 usec of CPU time to do so, and
  not consuming any memory (from the kernel's perspective; this is *not*
  fine-grained memory consumption).  The next lines indicate 12 usec were
  spent draining events and 10 usec expiring timers.  The last two lines
  indicate that the functions at lines 282 and 253 in conn.bro were
  executed, requiring 11 usec and 12 usec, respectively.

  Note #1: timings are just what rusage() reports, so not necessarily
  very accurate for small times.

  Note #2: there's a bug in tracking function line numbers that hasn't
  been ferreted out yet, so they're only roughly correct.

- The inactivity_timeout global has been split into tcp_inactivity_timeout/
  udp_inactivity_timeout/icmp_inactivity_timeout (Robin Sommer).  Using
  this, the default inactivity timeouts for UDP and ICMP have been changed
  from "no timeout" to 10 seconds.  This is needed because otherwise
  analyzing a stream of UDP or ICMP traffic generally gobbles up memory
  quickly and never recovers it; and there's seems little point in trying
  to consolidate long-lived-but-often-inactive UDP/ICMP streams.

- The new policy script cpu-adapt.bro is an extension to load-levels.bro
  (see CHANGES for 0.8a37 below) to adapt the packet filter based on the
  current CPU load. If the load is below cpu_lower_limit (default 40%),
  the load-level is decreased.  If it's above cpu_upper_limit (default
  90%), it's increased.  (Robin Sommer)
 
- The new policy script hand-over.bro can be used for a new running
  instance of Bro to smoothly take over operation from an old instance,
  i.e., it implements hand-over of state between two Bro instances when
  checkpointing (Robin Sommer). First, all persistent data (i.e. variables
  declared &persistent and connections for which make_connection_persistent()
  has been called) is transferred from the old instance to the new instance.
  Then the old instance terminates itself and the new one starts processing.

  The host from which we want to take over the state has to be added to
  remote_peers_clear (or remote_peers_ssl), setting hand_over to T. The
  host which is allowed to perform a hand-over with us has to be added
  with a port of 0/tcp and hand_over=T. An example for a handover between
  two instances on the same machine:

  @load hand-over
  redef remote_peers_clear += {
        [127.0.0.1, 47756/tcp] = [$hand_over = T],
        [127.0.0.1, 0/tcp] = [$hand_over = T]
        };

  (This interface may be subject to change in the future.)

- New script functions (Robin Sommer):

      function terminate()
          Terminates Bro via SIGTERM.

      function dump_config()
          Dumps Bro's full configuration into state_dir (one file per
          variable/type/function, etc.)

      function send_state(ip: addr, p: port)
          Send all of persistent state to the remote host.

      function set_accept_state(ip: addr, p: port, accept: bool)
          If accept is true, state sent by the given host will be
          accepted (default: false)

      function make_connection_persistent(c: connection)
          Declare the given connection state to be persistent (i.e.
          to be saved upon termination and exchanged by send_state).
          checkpoint.bro uses this to declare some services to be
          persistent by default.

      function is_local_interface(ip: addr): bool
          Returns true if the given address is assigned to a local interface.

- Printing of sets and tables now includes timestamps indicating when the
  element was added (Robin Sommer):

         ID did_ssh_version = {
            [129.187.20.9, F] = 1 @11/01-15:55,
            [212.144.77.26, T] = 2 @11/01-15:55,
            [141.84.116.26, T] = 10 @11/01-15:55,
            [217.232.245.249, T] = 1 @11/01-15:55,
            [217.235.217.149, T] = 1 @11/01-15:55,
            [129.187.39.13, F] = 2 @11/01-15:55,
            [129.187.208.139, F] = 1 @11/01-15:55,
            }

  The format may change in the future, and will probably be made an option.

- Similarly, you can print functions to get both a timestamp of the last
  time the given block was executed and a count of how often (Robin Sommer):

	  ID record_connection = record_connection
	  (@11/01-16:03 #6549)
	  {
	  id = c$id;
	  local_init = is_local_addr(id$orig_h);
	  local_addr = local_init ? id$orig_h : id$resp_h;
		    remote_addr = local_init ? id$resp_h : id$orig_h;
	  flags = local_init ? "L" : "";
	  if (remote_addr in neighbor_nets)
		  (@<never> #0)
		  flags = cat(flags, "U");

	  if ("" == flags)
		  (@11/01-16:03 #2110)
		  flags = "X";

	  is_tcp = is_tcp_port(id$orig_p);
	  ;
	  if (is_tcp)
		  (@11/01-16:03 #6549)
		  {
		  if (c$orig$state in conn_closed || c$resp$state in conn_closed
)
			  (@11/01-16:03 #4739)
			  duration = fmt("%.6g", c$duration);
		  else
			  (@11/01-16:03 #1810)
			  duration = "?";
	  [...]

- You can now specify numbers using hex constants, e.g., 0xabcd = 43981
  (Michael Kuhn and Benedikt Ostermaier).

- A new function, join_string_array(sep: string, a: string_array) concatenates
  strings in 'a' and inserts 'sep' between every two adjacent elements
  (Ruoming Pang).  E.g., join_string_array("", {"a", "b", "c"}) returns
  "a b c", and join_string_array("", a) is the same as cat_string_array(a).

- checkpoint.bro now makes some services persistent by default
  (Robin Sommer).

- The new_packet event now includes both the associated connection
  and a pkt_hdr describing the packet (Robin Sommer).

- The functions functions connect_ssl() and connect_clear() have been replaced
  by a single connect() function taking an additional parameter to
  differentiate the types (Robin Sommer).

- The new function stop_listening() unbinds the listen port (Robin Sommer).

- A new flag packet_filter_default says whether the Bro-level packet-filter
  will by default accept all or reject everything (Robin Sommer).

- Function calls can now be sent to remote Bro's, though there isn't yet
  an interface for accessing this from the script level (Robin Sommer).

- Bro now has an generalized internal framework for serializing objects
  and monitoring access to state (Robin Sommer).

- Better memory allocation accounting (Robin Sommer).

- A minor tweak to the output generated by statistics.bro.

- Improved localization of source code for functions in messages (but
  there are still some bug).

- Autoconf looks for -ltermcap (Robin Sommer).

- Fixes for bugs in the management of table expiration values (Chema Gonzalez).

- A bug in printing "void" values has been fixed (Chema Gonzalez).

- -t bug fixed (Chema Gonzalez).

- A bug has been fixed in which sometimes "expression value ignored"
  was erroneously generated.

- A bug with packet_contents and UDP packets with checksum errors
  has been fixed (Ruoming Pang).

- A memory leak in packet timestamp sorting via packet_sort_window
  has been fixed (Ruoming Pang).

- A bug has been fixed in expanding NULs when printing strings (Ruoming Pang).

- Bug fixes for extracting connection contents via contents.bro (Ruoming Pang).

- Bogus error message "Can't install default pcap filter" when using -F
  removed.