[Bro] Bro 1.5 release now available

[Vern Paxson]

Bro release 1.5 is now available from:

	ftp://bro-ids.org/bro-1.5.tar.gz

This release includes new functionality as well as numerous refinements
and fixes, per the appended changelog entries.

Previous releases are available at http://www.bro-ids.org/download.html .

		Vern


-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+


1.5 Wed Dec 16 21:28:47 PST 2009

- Bro now comes with a new framework, BroControl, for managing an
  operational Bro setup, including support for installation, configuration,
  and maintainance tasks such a log archival and mail notification.  The
  framework transparently supports both traditional standalone setups as
  well as cluster installations in which multiple Bro boxes coordinate to
  analyze a high-volume network link.

  See aux/broctl/README for more information about BroControl.

  Note, BroControl supersedes the older BroLite system, which is no longer
  supported and has been deprecated for a while now.

- Numerous adjustments to DPD = dynamic protocol detection (Robin Sommer):

    o The Analyzer::ProtocolViolation?() method can now be passed the
      offending data (which POP3, SMTP, and FTP now do).  This information
      is added to the "reason" string passed to the script level.

    o SMTP now more accurately reports violations.

    o FTP stops processing when client & server successfully negotiate
      an AUTH scheme (leading to subsequent encryption).

    o Analyzer::ProtocolViolation() is virtual, and
      TCP_ApplicationAnalyzer() overrides it to not report violations
      for any partial connections, because very likely these arise just
      due to the analyzer getting confused.

    o TCP::IsPartial() returns true if any side did not start with
      a SYN packet (used to be just be for the originator).

    o The connection_state_remove handler in conn.bro now has a higher
      &priority so that other handlers for the same event can use
      determine_service() and see any changes it performs.

    o DynDisable:max_volume specifies a volume limit (default 10K).
      Once a connection exceeds this limit, further protocol
      limitations will neither raise ProtocolViolation notices nor
      cause the analyzer to be disabled.

    o The event engine no longer raises protocol_violation events for
      TCP connections which had gaps, as these have proven too unreliable.
      (Note that, ideally, the *analyzers* should avoid reporting
      protocol_violations when they can't reliably parse a connection
      anymore after a gap; but many don't.)

- A set of new script functions provide support for incrementally computing
  MD5 checksums (Seth Hall).

	md5_hash_init(index: any): bool
		Initializes an incremental hashing instance.  "index" is
		a value of arbitrary type, used to identify this particular
		instance (you can have multiple concurrent instances by
		using different index values).  Returns T on success,
		F on failure (such as the index is already in use).

	md5_hash_update(index: any, data: string): bool
		For the given hashing instance, updates the hash
		based on the given data.  Returns T on success, F on
		failure (such as the index has not been initialized).

	md5_hash_finish(index: any): string
		Returns the MD5-printable hash for the given index
		and terminates the instance, or the string "" if the
		index was not active.

- Bro now supports a believed-to-be-robust mechanism for estimating the
  proportion of traffic that it failed to capture ("measurement drops"),
  which can arise due to overload in either Bro itself, the kernel's
  packet filter, or problems with the link tapping mechanism (Vern Paxson).
  The event engine can generate estimates for either live traffic or what
  was previously recorded in a trace file, though traces subject to some
  forms of selective omission (such as skipping over parts of a connection
  to reduce storage) can lead to erroneous values.

  The estimates are based on observing gaps in TCP data streams, and
  come in two forms: the rate at which such gaps appear, and the relative
  volume of data missing due to the gaps.  (We've found however that the
  volume-based estimator is not robust due to occasional packets with
  incorrect sequence numbers, so this estimator is off by default.)

  The easy way to get the estimates is to load capture-loss.bro.
  By default, it generates a CaptureLossSummary notice upon Bro's exit,
  which can look like:

	1130222759.344066 CaptureLossSummary estimated rate = 0.00089124 / 0.000970997 (events/bytes)

  If the estimated loss is none, however, it suppresses this notice,
  unless you redef CaptureLoss::summary_if_none to T.

  You can also get finer-grained access by defining a "gap_report"
  event handler and redef'ing gap_report_freq to a non-zero interval
  (such as "10 sec").  This event allows you to pinpoint regions in
  time that exhibit significant capture loss.  See capture-loss.bro
  for an example of a handler for this event.

  Finally, these changes include a number of fixes to Bro's
  ack_above_hole/content_gap analysis, which is now significantly
  more robust.

- GeoIP support now supports ASN lookups via the built-in
  function lookup_asn(a: addr): count (Scott Campbell and Seth Hall).

- The GeoIP built-in's lookup_location() and lookup_asn() now
  support IPv6 (Seth Hall).  Note, the current GeoIP distribution
  doesn't include any IPv6 databases, so for now these won't succeed,
  but the hooks are in place for when databases become available.

- lookup_location() now falls back back to the country database if
  the city database isn't available (Seth Hall).

- The new SuccessfulPasswordGuessing Notice is generated when a host
  has been seen attempting password guessing (currently only for FTP
  sessions) and then successfully logs in (Royal Chan).  You can control the
  threshold for such reports in terms of how many attempts the host must
  have made by redef'ing the variable password_guessing_success_threshhold,
  which defaults to 20.

- The new script http-detect-passwd.bro analyzes the Web items returned
  for fetches that appear to be accessing the passwd file (Akhil Dhar).
  It generates a PasswordFullFetch Notice if it appears that the item
  includes a full password file, and PasswordShadowFetch if it looks like
  a shadowed password file.

- The new built-in

	system_env(cmd: string, env: table[string] of string)

  works like system(), but puts the table entries into the environment
  before invoking the command (Robin Sommer).  Each <index> in the table
  creates an environment variable of the form "BRO_ARG_<index>", whose
  value is the corresponding table entry.

- The new script function

	execute_with_notice(cmd: string, notice_info)

  executes "cmd" with an environment containing the fields of the
  notice_info, i.e., the information associated with a Notice (Robin Sommer).
  Per the new system_env() function above, the environment variables appear
  as "BRO_ARG_<tag>", where <tag> is the field tag as it appears in
  notice.log when you enable use_tagging.

- The new built-in enable_raw_output(file) acts the same as
  the attribute &raw_output (Seth Hall).

- The new built-in file_opened(f: file) event is generated any time Bro
  opens a script-level file (Justin Azoff).  You can use this, for example,
  if you want to ensure that a given file has a prelude in it such as
  human-readable headers, even when the file is rotated.

- The notice_info record has a new field

	aux: table[string] of string &optional

  which you can use for information specific to a given type of notice
  (Robin Sommer).  Entries in $aux appear as "aux_<index>" tags in notice.log.

- Another new notice_info record field is the boolean do_alarm (default=T),
  which, if set to F, overides a notice action otherwise specifying to
  generate an alarm (Robin Sommer).  In other words, if do_alarm is F, no
  alarm will be generated independent of the notice action.

  This is a work-around for the fact that we can't specify more than one
  action.  In particular, we couldn't NOTICE_DROP but then *not* alarm,
  which we now can by returning NOTICE_DROP yet setting do_alarm to F.

- The notice_info record field $dropped now appears in the tagged output
  format if true (Robin Sommer).

- NOTICEs relating to scan detection now no longer include the connection
  that triggered the notice, as it really doesn't contain any useful
  information, given that the particular trigger simply depends on the
  detection algorithm and its parameters (Robin Sommer).  However, we do
  explicitly set $p (port number) in the notice, and also $n with the
  number of attempts.

- drop.bro now hardwires a Catch-and-Release redrop after seeing one
  connection from a previously-dropped-but-already-released host
  (Robin Sommer).

- drop.bro now provides some new hooks (Robin Sommer):

	event address_dropped(a: addr)
		Generated when an address has been dropped.

	event address_restored(a: addr)
		Generated when connectivity to an address has been restored,
		such as using the Catch-and-Release mechanism.

	event address_cleared(a: addr)
		Generated when an address that was dropped in the past is
		no longer being monitored looking for new connections
		(as part of the Catch-and-Release mechanism).

- The new built-in function

	hexdump(data_str: string) : string

  returns a hex dump representation of the given input data (Christian
  Kreibich).  The dump renders 16 bytes per line, with hex on the left and
  ASCII (where printable) on the right.

- Bro's notion of when a TCP connection begins now dastes to the first
  instance of an initial SYN packet seen, rather than the last (Gregor Maier).

- The Time Machine script tm-contents.bro now generates

	event contents_saved: event(c: connection, orig_file: string,
					resp_file: string)

  when the content of a connection has been completely saved to disk
  (Robin Sommer).

- The mime.bro script now exports the MIME header callback table, and also
  marks it as &redef'able so you can modify its entries (Matthias Vallentin).
  The mime_log file is also now exported.

- A new signature file, policy/sigs/http-bots.sig, contains signatures
  to detect some of the current HTTP based controlled bot families (Seth Hall).

- The signature engine's HTTP pattern matching has been fixed (Seth Hall)
  to align with the documentation at:

  http://www.bro-ids.org/wiki/index.php/Reference_Manual:_Signatures#Content_conditions

  In particular, the content condition "http" is now referred to as
  "http-request" (though "http" still works for backward compatibility),
  "http-request-header" and "http-reply-header" now provide access to
  headers seen in only one direction, and similarly for "http-request-body"
  and "http-reply-body".  (This latter is still accessible as "http-body"
  for backwards compatibility.)

- The new script variable max_remote_events_processed: count (default 10)
  sets a limit on the number of remote events processed in each round,
  before tending to other inputs (Robin Sommer).

- If you set the new script variable dump_used_event_handlers to T,
  then on startup Bro dumps out all of the event handlers that the
  loaded set of scripts can invoke (Matthias Vallenti).

- Summaries for DNS PTR scanning now use a separate Notice,
  DNS_PTR_Scan_Summary, rather than overloading DNS_PTR_Scan (Robin Sommer).

- scan.bro now provides a table skip_dest_server_ports: set[addr, port]
  which lists servers (defined as an address and a port) excluded from
  scan detection computations (Craig Leres and Jay Krous).

- When redefining values on the command line directly (using var=value),
  quotation marks are now implicit only if "var" is a variable of type
  string (Christian Kreibich).  This allows other string-like values
  (such as enum's) to be passed as well.

- scan.bro now explicitly loads conn.bro so that it can itself
  be loaded independently (Robin Sommer).

- login.bro depends on scan.bro (because of tracking authentication
  "scans"), so now it explicitly loads it (Vern Paxson).

- UDP_datagram_length_mismatch is now by default flagged just once per
  originating host rather than once per connection, as it can generate
  tons of messages (Vern Paxson).

- Removed now-long-boring flagging of access to Solaris "listen"
  service as "hot" (Vern Paxson).

- Removal of libedit, since libreadline provides similar functionality
  (Christian Kreibich).

- Added scripts missing from distribution: dce.bro, ncp.bro, and smb.bro
  (Vern Paxson).

- ssh.bro now exports ssh_ports (Seth Hall)

- A number of improvements to inter-Bro communication (Robin Sommer).

 (1) Remote communication now no longer includes location information for
 serialized objects; that removes quite a bit of redundacy from the network
 traffic.
 
 (2) The new option 'remote_check_sync_consistency" disables the cross-check
 on the receiving side of &synchronized state of whether the current value
 of a variable has the value expected by the sender. Transmitting the
 original values in addition to the updates generates quite a bit CPU &
 network load in some cases (in particular, a table of tables). The default
 for remote_check_sync_consistency is off, and so far that in particular
 seems to reduce the proxy's load quite a bit.

 (3) Complete overhaul of the internal caching of serialized objects.  The
 objective of the caching is avoid retransmitting already sent values over
 and over again. It turns out, however, that some objects are very stable
 and hardly change or get replaced (e.g., Bro types); while other change
 all the time and are hardly reused some time later (e.g., Vals).  Now
 we maintain *two* caches independently for these types of objects; one
 with a low turn-over one and another with a high one.  This should reduce
 CPU load on both sender and receiver sides.
 
 The new scheme is only used if both communicating Bros support it; with
 older Bros, as well as with Broccoli, we continue using the old scheme.

- Some reworking of remote printing (Robin Sommer), as follows.  Bro now
  uses a new interprocess message rather than print_hook events, to better
  manage buffering and associated load (these can produce failures depending
  on system configuration; see remote.log).  A number of timeouts and
  buffer sizes have been tuned.  Internally, EINTR errors are now treated
  separately from EAGAIN.  Finally, even with remote_check_sync_consistency=F,
  one type of consistency check was still being done; this is no longer
  the case.

- The DNS analyzer now generates events (dns_query_reply/dns_rejected)
  for replies with zero questions (Robin Sommer).

- Perftools support for incompatible changes in the 1.0 API (Robin Sommer).

- Rearranged (generally reducing, though not always) some state timeouts
  associated with scan detection (Robin Sommer).  In addition, when a
  scanning address crosses ignore_scanners_threshold (meaning that it will
  be ignored from now on anyway), it gets discarded from all state-tracking
  tables.  Finally, the ignore_scanners_threshold now applies all kinds
  of scans, not just address scans.

- Substantial Broccoli updates, including a new initialization requirement
  that breaks backward compatibility, support for enqueueing serialized
  event data for transmission, and OpenSSL threadsafe initialization.
  See aux/broccoli/ChangeLog for details (Christian Kreibich, Robin
  Sommer, and Matthias Vallentin).

- Broccoli hashtable optimisation. See aux/broccoli/ChangeLog for
  details (Christian Kreibich & Matthias Vallentin).

- Broccoli memory leak fixed, see aux/broccoli/ChangeLog for details
  (Christian Kreibich).

- Broccoli: updates to bropipe tool (Steve Chan and Robin Sommer).

- Bug fixes for Broccoli Python bindings (Robin Sommer and Matthias Vallentin).

- Fixed nasty bug due to module scoping that completely kept stepping-stone
  detection from working (Vern Paxson).

- A serious bug in the packet sorter has been fixed (Robin Sommer).

- Bug fix for extra NULs getting embedded in escaped strings (Seth Hall).

- Bug fix for HTTP messages that use "Connection: close" rather than length
  headers, which yielded erroneous reassembled messages with \r\n's when
  only \n's were present (Bernhard Ager).

- Fix for reporting on ICMP flows that are expired from the flow table
  (Vern Paxson).  Previously there was a race condition if the flow
  was flushed prior to its summary timer expiring.

- The -l option (list the scripts that Bro loads) now correctly prints
  scripts loaded by the prefix mechanism, and uses indentation to indicate
  the load hierarchy (Robin Sommer).

- A bug has been fixed (really, worked around) in drop.bro that prevented
  dropped addresses from being properly restored (Robin Sommer).

- Fixes for deadlocking problems in the Broccoli protocol. See
  aux/broccoli/ChangeLog for details (Christian Kreibich & Robin Sommer).

- Bug fix for DNS analyzer on 64-bit machines (Gregor Maier).

- Bug fix for asynchronous DNS lookups to prevent some successful lookups
  being reported as timed out (Robin Sommer).

- Bug fix for tracking line numbers associated with compound statements
  (Po-Ching Lin).

- Fix for a rare condition in which the main Bro process couldn't kill
  its child process (Robin Sommer).

- Fix for file rotation when the underlying file is deleted before the
  timer expires (Robin Sommer).

- Fix for potential crash when communication connections break down,
  and also for releasing cached objects (Robin Sommer).

- Fix for default table entries computed by function invocation to not
  cache previous results (Robin Sommer).

- Fix for Bro's internal DNS resolution (Scott Campbell and Robin Sommer).

- Portability fix for DAG packet capture (Gregor Maier).

- Portability fix for --enable-brov6 (Robin Sommer).

- Portability fixes for FreeBSD (Vern Paxson).

- A work around for new_packet() crashing on IPv6 packets (Vern Paxson).
  For now, IPv6 packets are skipped.  Also, for fragments the event handler
  is now only called for the fully reassembled packet.

- The new configuration option --disable-nbdns supports disabling non-blocking
  DNS at configure time (Sean McCreary).  Note, there are some known problems
  with it in some environments.

- A number of configuration fixes and enhancements (Christian Kreibich
  and Robin Sommer).

- Consistency nit for the configuration process (Seth Hall).

- A number of reference-counting and other memory management fixes
  (Robin Sommer).

- Bug fix for inter-Bro communication lockup (Seth Hall and Robin Sommer).

- Bug fix for computing TCP payload length in new_packet event (Lothar Braun).

- Bug fix for sending boolean True values via Broccoli (Seth Hall).

- make distcheck fix to clean up .bif.bro files (Christian Kreibich).

- Bug fix for DPD's recognition of SSLv2 connections (Seth Hall).

- Bug fix for &default for tables indexed by subnets (Seth Hall).

- A bug has been fixed that could crash Bro when you called get_event_peer()
  after a remote connection had already disppeared (Robin Sommer).

- Introduced a work-around for crashes that occur when Bro exits
  due to handling a signal (Robin Sommer).

- Bug fix for checkpoint.bro - don't schedule timers for times that
  aren't actually in the future (Robin Sommer).

- Hostname formatting fix for anon.bro (Fabian Schneider).

- Bug fix for redundant .log extension in Time Machine log file
  (reported by CS Lee).

- Removed now-outdated special-casing of Linux reporting of packet filter
  statistics (Peter Wurzinger and Robin Sommer).

- A number of memory leaks fixed (Robin Sommer).

- Addressed warnings from newer versions of g++ (Robin Sommer and Vern Paxson).

- Fixed an invocation issue in the ca-create script that prevented it from
  working with recent OpenSSL versions (Craig Leres & Christian Kreibich).

- Comment fixed in drop-adapt (Justin Azoff).

- Duplicate code removed from Val (Seth Hall).