new bro "CURRENT" release - 0.8a48

[Vern Paxson]

An updated "CURRENT" version of Bro is now available from

	ftp://ftp.ee.lbl.gov/bro-pub-0.8-current.tar.gz

I've appended the changes between it and the last "CURRENT" version (0.8a37).

		Vern


-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+


0.8a48 Tue Oct 21 15:56:13 PDT 2003

- There is now a mechanism in place for multiple Bro's to communicate with
  one another via sockets (Robin Sommer).  *This is quite experimental at
  this point* and may have significant bugs and/or need significant
  enhancements.

  By loading listen-clear.bro or listen-ssl.bro, an instance of Bro starts
  to listen on a TCP port.  The first of these listens for unencrypted
  connections and the second for connections encrypted via OpenSSL.  To
  connect to a listening Bro, you load remote-clear.bro or remote-ssl.bro.
  For this connection, you specify which events you want to receive by
  giving a regular expression (e.g.  "/http_*/" for all HTTP events),
  although only those events for which you have defined a local handler
  are actually requested.  Communication is uni-directional in the sense
  that for a certain connection the events go only from Bro-A to Bro-B but
  not from B to A (you could set up a second connection for the other
  direction, though).

  The OpenSSL certificate used to authorize remote  Bro's is specified in
  the script variable "ssl_ca_certificate" and the private key and certificate
  for the local Bro via "ssl_private_key".

  If Bro A connects to Bro B, by default it sends over its capture_filter.
  But Bro B uses it only if it has loaded remote-pcap.bro.  This is the
  beginning of structuring inter-Bro trust mechanisms.  It is done via two
  new events, remote_capture_filter(ip: addr, p: port, filter: string) and
  send_capture_filter(ip: addr, p: port, s: string) : bool.

  The implementation forks a second process which does the socket
  communication, so that the main process should not be affected too much.
  The two processes communicate via a pipe.

  You can call is_remote_event() to determine whether the event currently
  being handled was originated by a remote Bro (if T) or the local Bro
  (if F).

  If a connection with a remote Bro terminates (for whatever reason), Bro
  may try to reconnect automatically.

  A new function, get_event_source(), returns a record event_source
  describing the source that raised the last event.

  See doc/ssl.txt for an explanation of how to create the keys/certificates.   

- A fledgling Gnutella analyzer has been contributed (Mark Allman).
  It generates the following events:

	event gnutella_text_msg(c: connection, orig: bool, headers: string)
	event gnutella_binary_msg(c: connection, orig: bool, msg_type: count,
					ttl: count, hops: count, msg_len: count,
					payload: string, payload_len: count,
					trunc: bool, complete: bool)
	event gnutella_partial_binary_msg(c: connection, orig: bool,
						msg: string, len: count)
	event gnutella_establish(c: connection)
	event gnutella_not_establish(c: connection)
	event gnutella_http_notify(c: connection)

- Bro now supports a secondary channel for acquiring packets (Chema Gonzalez).
  You access it by redef'ing the new global "secondary_filters", adding
  table[string] of event(filter: string, pkt: pkt_hdr).  The string
  specifies a tcpdump filter; anytime a packet matches the filter
  (including packets that would *not* otherwise be matched by
  capture_filter), then the given event handler is invoked.

  For example,

	  redef secondary_filters += {
		  ["tcp[13] & 7 != 0"] = rst_syn_fin_flag,
	  }
  
  will invoke rst_syn_fin_flag() anytime a TCP packet is seen for
  which the SYN/FIN/RST bits are non-zero.  The event handler will
  be passed the string "tcp[13] & 7 != 0" (so it can tell which
  of possibly multiple filters matched) and a pkt_hdr value, which
  includes the IP header and, if present, the TCP, UDP or ICMP header.

  Another example, and what motivated the addition, is:

	  redef secondary_filters += {
		["ip[10:2] & 0xffc == 0x398"] = sampled_1_in_1024_packet,
	  }

  which will invoke sampled_1_in_1024_packet() any time the given
  10 bits in the IP checksum match the pattern 0x398.  If the checksum
  field is uniformly distributed then this roughly corresponds to
  1-in-1024 random sampling.  (Chema has also developed BPF additions
  to support true random sampling.)

  See policy/secondary-filter.bro for examples.

- Bro now does a much better job of keeping track of how much memory
  has been allocated for different structures (Robin Sommer).

  This includes more accurate computations for global_size().

  In addition, if you redef "statistics_interval" to be a non-zero time
  interval, then with that periodicity a summary of memory usage (including
  memory used by event engine components) is dumped to the file
  "statistics_file".  In addition, at this point a "do_statistics" event
  is generated.  You can also call the new built-in statistics_update()
  to generate memory statistics on demand.

  The above structure is likely to change in the future.  statistics_interval
  will probably go away, to be replaced by either explicit calls to
  statistics_update() (which you can do on a timer if you like by using
  "schedule"), or by a built-in function that returns a record of all
  the statistics, that you can then format however you want.

- A major memory leak in HTTP analysis has been fixed (Ruoming Pang).

- New attributes &rotate = <interval expression> and
  &postprocessor = <string expression> can be associated with a file
  variable in order to specify how often the file should be rotated to a
  new filename on disk, and, when rotation occurs, the name of a shell
  program to run on the now-older version as a postprocessor (Robin Sommer).

- Similarly, log_postprocessor and log_rotate_interval specify the default
  values for files.  Unless redef'd, these themselves default to the empty
  string (= no postprocessing) and 0.0 seconds (no rotation).  (Robin Sommer)

- A new attribute, &encrypt, applies to variables of "file" type.  It specifies
  that the version on disk should be encrypted, using either the key specified
  as the value of the attribute, or, if no value is specified, using the
  value of the new script variable log_encryption_key.  The key is an OpenSSL
  public key; it's used to then embed a Blowfish session key.  (Robin Sommer)

  A new utility, aux/bdcat/bdcat ("Bro decrypt cat") can be used to decrypt
  the files. 

- The internal structure of TCP analysis has been significantly altered.
  Previously, TCP_Endpoint tracked endpoint state and TCP_EndpointContents
  (derived from it) did stream reassembly.  These have now been separated;
  TCP_Endpoint still tracks endpoint state, but TCP_EndointContents has
  become TCP_Contents, and is only loosely coupled with TCP_Endpoint.
  The reason for this change is to lay the groundwork for (1) applying
  an application analyzer to a connection after several packets for
  the connection have already been read, and (2) applying *multiple*
  application analyzers to a single connection.

- Bro now supports the universal hashing if USE_UHASH is defined
  (Ruoming Pang).  Universal hashing is a lighter-weight alternative
  to MD5 hashing that retains the property of being very difficult
  for an attacker to guess.  It comes in two flavors, a 32-bit
  version (which you get with USE_UHASH) and a faster 16-bit version
  (which you get if you also define USE_NH).  Bro will likely switch
  to using these by default in the near future, as their performance
  gain over MD5 is significant.

- New built-in functions srand() and rand() provide access to the
  usual C-style random number seeding & generation (Chema Gonzalez).

- You can now specify server/client addresses to leave in the clear in
  IP address anonymization (via the built-in variables preserve_orig_addr
  and preserve_resp_addr). Correspondingly, the built-in function for
  IP anonymization now takes a parameter to specify the type of the address
  (orig, resp, or other), instead of the method of anonymization
  (seq, random, prefix-preserving).  (Ruoming Pang)

- Trace anonymization now has prelminary support for handling TCP options
  via the new event "tcp_option" (Ruoming Pang).  It is only invoked
  if skip_event_tcp_option is F (it defaults to T).

- A new event, tcp_segment, is similar to the event "packet_content"
  but provides more information: is_orig (directionality), sequence
  number, and acknowledgement number (Ruoming Pang).

- ./configure finds OpenSSL if it's in some standard location.  Otherwise,
  you may specify it --with-openssl=<path>.  If OpenSSL is not available,
  Bro compiles cleanly without and gives warnings if a script tries use SSL.
  (Robin Sommer)

- The internal links in manual/entire.html have been fixed so they
  now work (Chema Gonzalez).

- A new policy script, blaster.bro, detects instances of the W32.Blaster
  worm (Robin Sommer).

- Signature files (for input to the signature engine) now reside in
  policy/sigs/*.sig.  This directory is now on the default $BROPATH.

- sig.ex.ssl-worm.bro and sig.ex.web-rules.bro have been updated
  to reflect changes in keywords (Robin Sommer).  They've been
  renamed to ex.ssl-worm.sig and ex.web-rules.sig and reside
  in policy/sigs/, per the above.

- The module facility has been changed to have its scope limited to
  the current file plus any additional files that are automatically
  processed based on its name plus $BRO_PREFIXES.

- As an experiment, ftp.bro has been modified to a style that includes
  using "module FTP".  Likely other policy scripts will be converted
  in the near future, and their variable names changed accordingly
  (e.g., "ftp_hot_files" will become "FTP::hot_files").

- The new "match" expression has been modified to allow any yield type
  rather than just strings.  It is likely to change significantly again
  soon.

- Iterating over multi-dimensional tables/sets now works (Robin Sommer).
  For example:

    const remote_peers_ssl : table[addr, port] of Peer &redef;
    [...]
    for ( [ip, p] in remote_peers_ssl )
        connect_ssl(ip, p, remote_peers_ssl[ip, p]$retry); 

- Checkpointing of persistent state on SIGHUP now happens via bro.init
  (Robin Sommer).  Not tested.

- fmt() now supports %x for hexadecimal formatting (Chema Gonzalez).

- alert.bro logs the source for remote alerts; by redefining the new
  "event_source_description: string" locally, you can specify how a
  Bro is to refered to on the remote side.  (Robin Sommer)

- software.bro now tracks HTTP clients, too (Robin Sommer).  This
  will be extended in the near future.

- Some FreeBSD 5.X porting tweaks (Sergey Osokin).