new Bro pre-release available - 0.7a90

[Vern Paxson]

A new Bro pre-release is now available from:

	ftp://ftp.ee.lbl.gov/.vp-bro-pub-0.7a90.tar.gz

The main theme of this release is better state management, especially for
use in high-volume sites.  There are now a number of ways to track the
resources consumed by Bro while it runs, and to reduce the amount of memory
Bro consumes.  Adding

	@load print-resources

will log a summary of the resources consumed by Bro when it exits, and

	@load reduce-memory

will change a number of default values in order to significantly diminish
the amount of memory Bro requires.

Other ways to lower the required memory:

	- The global "maintain_http_sessions" now controls whether
	  http.bro tracks multiple HTTP connections between the same
	  hosts as a single HTTP session.  Doing so costs a considerable
	  amount of state (so set to F for reducing the memory impact
	  of HTTP analysis).

	- The global "suppress_scan_checks", if true, turns off address
	  and port scan detection.  (You can't achieve this by simply
	  not @load'ing scan.bro, because it's loaded by some of the
	  default policy scripts.)  Turning it off can save a lot of memory.

	- Note, the ident.bro is also expensive in terms of state-holding.

A summary of other changes with respect to the last snapshot:

	- New library functions:

		- resource_usage() returns a record detailing real time,
		  CPU time, memory, other getrusage info, and the current
		  and maximum number of TCP/UDP/ICMP connections, and
		  timers and fragments.

		- val_size() returns the size in bytes needed to represent
		  a given value (which can be a record, a table, or a
		  simple constant, for example).  It's not fully accurate
		  but is in the right ballpark.

		- global_sizes() return a table mapping every global
		  variable to its size (per val_size()).  Useful for
		  tracking which ones are growing large over time.

	- You can now control a number of timers related to Bro's connection
	  management.  Setting them to lower values generally decreases the
	  amount of state Bro has to keep (see reduce-memory.bro), though
	  this can also make it easier for an attacker to evade detection:

		tcp_SYN_timeout: interval
			Controls how long Bro waits after seeing the
			beginning of a connection (whether due to a SYN
			or not; the timer is misnamed) before checking
			whether it elicited any reply.

		tcp_session_timer: interval
			After a connection has closed, wait this long for
			further activity before checking whether to time
			out its state.

		tcp_connection_linger: interval
			When checking a closed connection for further
			activity, Bro should consider it inactive if there
			hasn't been any for this long.  It also complains
			if the connection is reused before this much time
			has elapsed.

		tcp_attempt_delay: interval
			Bro waits this long upon seeing an initial SYN
			before timing out the connection attempt.

		tcp_close_delay: interval
			Upon seeing a normal connection close, Bro flushes
			state after this much time.

		tcp_reset_delay: interval
			Upon seeing a RST, Bro flushes state after this
			much time.

		tcp_partial_close_delay: interval
			Bro generates a connection_partial_close event this
			much time after one half of a partial connection
			closes, assuming there has been no subsequent
			activity.

		non_analyzed_lifetime: interval
			If a connection belongs to an application that you
			aren't analyzing, Bro times it out after this
			interval.  If 0 secs (default), then it doesn't
			time it out.

		dns_session_timeout: interval
		ntp_session_timeout: interval
		rpc_timeout: interval
			Bro waits this long before timing out a DNS/NTP/RPC
			request.

		max_timer_expires: count
			The maximum number of timers to expire after
			processing each new packet.  The value trades off
			spreading out the timer expiration load with
			possibly having to hold state longer.  A value of 0
			means "process all expired timers with each new
			packet".

	- Two new flags control other facets of Bro's connection management,
	  and thus state-holding:

		partial_connection_ok: bool
			Whether Bro should analyze connections for which
			it doesn't see the beginning, only the middle.
			This can be very expensive to do in the face of
			stealth-scanning, which looks like a bunch of
			partial connections.

			Note, the HTTP analyzer has been modified to
			now always skip partial connections.  This should
			instead be user controllable.

		tcp_SYN_ack_ok: bool
			If true, Bro instantiates connection state when
			it sees a SYN ack but not the initial SYN (even
			if partial_connection_ok is false).  The intent
			behind this knob (which is not well tested) is
			to allow you to filter out initial SYNs and only
			react to SYN acks.  This keeps Bro from holding
			state during SYN scans and SYN flooding, except
			for when the destination responds.

	- Some other miscellaneous thresholds that you can now modify
	  from your policy script:

		heartbeat_interval: count
			How often to generate net_stats_update() events.
			This timer really isn't needed any more, since
			you can use "schedule" to achieve the same effect.

		tcp_storm_thresh: count
			If Bro sees this many FINs/RSTs in a row, it
			flags them as a "storm".

		tcp_storm_interarrival_thresh: interval
			The FINs/RSTs must come with this much time or less
			between them.

	- The state management for line-oriented applications like HTTP
	  requests has been improved.

	- The HTTP analyzer now expands %hex sequences.  If anyone has
	  a Unicode expander to contribute, that'd be terrific.

	- The Code Red detection is more robust (fewer false positives).

	- A new redefinable variable, skip_services, lists applications
	  that should not be analyzed for purposes of detecting scans.
	  (Default: ident)  The point of having this (poorly named) hook
	  is so that code-red.bro can add "http" to it, to avoid reporting
	  the zillions of Code Red scans that a site can see.

	- Bro now matches regular expressions using lazy DFA evaluation.
	  The upshot of this is (1) it no longer maintains .bro-RE-cache.v1,
	  (2) it starts up fast regardless of whether you've added new
	  regular expressions, (3) you can afford to add lots of regular
	  expressions, and (4) it's actually a bit faster.
 
	- The list of "hot_ids" has been expanded with a number of other
	  common root-privileged accounts.

- Vern