[Bro] crash with std::bad_alloc

[Peter Wurzinger]

Robin Sommer wrote:
> On Wed, Nov 05, 2008 at 18:22 +0100, Peter Wurzinger wrote:
> 
>> terminate called after throwing an instance of 'std::bad_alloc'
>>    what():  std::bad_alloc
> 
> That sounds like Bro is running out of memory. What's the process'
> size just before it crashes and how much memory does the machine
> have?

I reran my code using profiling.bro. The memory consumption continuously 
increased, and the last lines before crashing were:

Memory: total=3126520K total_adj=3116888K malloced: 2878549K
Run-time: user+sys=2861.5 user=2528.8 sys=332.7 real=3390.1
Conns: total=915256 current=19795/19795 ext=0 mem=0K avg=0.0 table=0K 
connvals=0K
ConnCompressor: pending=0 pending_in_mem=0 full_conns=0 pending+real=0 
mem=0K avg=nan/nan
Conns: tcp=7431/8335 udp=11845/20278 icmp=519/783
TCP-States:        Inact.  Syn.    SA      Part.   Est.    Fin.    Rst.
TCP-States:Inact.                          16      159     2       3
TCP-States:Syn.    106                     243             147     14
TCP-States:SA      1                       185             64      1
TCP-States:Part.   38                      1436    101     175     9
TCP-States:Est.                                    2052    1023    22
TCP-States:Fin.    3                       258     1014    261     6
TCP-States:Rst.    8                       12      63      9
Connections expired due to inactivity: 697053
Total reassembler data: 134947K
Timers: current=30215 max=34256 mem=1652K lag=0.00s
          ConnectionDeleteTimer = 180
          ConnectionInactivityTimer = 19688
          NetworkTimer = 1
          ScheduleTimer = 241
          TableValTimer = 2
          TCPConnectionAttemptTimer = 238
          TCPConnectionExpireTimer = 9859
          TCPConnectionResetTimer = 6


>> matches anymore, still the error occurs. My signature file is approx
>> 100Kb, contains more than 600 signatures, and all of them look like:
> 
> If it's indeed memory exhaustion, then it looks like either a memory
> leak in the signature engine or a general problem of handling the
> many regexps. Generally, the engine can consume quite a bit of
> memory due to the DFAs it builds incrementally. How do your regexps
> look like? Do they contain many unanchored subparts (e.g.,
> "foo.*bar")? 

Yes, '.*' is massively used. Actually, that is the only regexp feature 
that is used. The patterns generally look like: 
".*byte_seq1.*byte_seq2.*byte_seq3.*"

Peter.