[Bro] question: memory requirements of tables

[Christoph Goeldi]

hi

sorry, but i have already another question.

i noticed that my bro (version 0.9a8.17) eats up my memory.
even with the reduce-memory.bro script it needed up to 2GB
or more after a while and swapped all the time.
when we reduced the traffic the memory usage decreased only
a little bit, but remained very high.

because we use &*_expire tags and scheduler to keep tables
small, we tried to reproduce our problem with a small script:

> ### memory_test.bro
> 
> # check all 10 secs if table entries are expired
> redef table_expire_interval = 10 secs;
> 
> # When expiring, don't check more than this many table entries at a time.
> redef table_expire_size = 10000;
> 
> global test_table: table[addr] of count &create_expire=30secs;       
> 
> 
> event print_table () {
>         print "size of test_table in bytes", val_size(test_table);
>         print "length of test_table", length(test_table);
>         print "test_table", test_table;
>         schedule 1 min { print_table() };
> }
> 
> event bro_init () {
>         schedule 1 min { print_table() };
> }
> 
> event udp_request(u: connection) {
>     test_table[u$id$resp_h] = 0;
> }

the following is the output of our script when doing some udp scanning.
this logs shows that all entries in the table have been deleted, but the
table uses still a lot of memory! what happened.

we tried different constellations and noticed that it doesn't matter
if the table entries have been deleted with the "delete" function or
the expire tag. we also tested it over a longer period (let it running
through the night) but the memory will not be deallocated.

> size of test_table in bytes, 304
> length of test_table, 0
> test_table, {
> 
> }
> size of test_table in bytes, 760
> length of test_table, 3
> test_table, {
> [x.x.228.20] = 0,
> [x.x.229.229] = 0,
> [x.x.229.5] = 0
> }
> size of test_table in bytes, 768
> length of test_table, 1
> test_table, {
> [x.x.229.229] = 0
> }
> size of test_table in bytes, 840
> length of test_table, 1
> test_table, {
> [x.x.229.229] = 0
> }
> size of test_table in bytes, 1675832
> length of test_table, 12545
> test_table, {
> [93.172.138.23] = 0,
> [60.114.97.75] = 0,
> [4.103.6.3] = 0,
> [51.164.85.16] = 0,
> [85.83.252.215] = 0,
> ...
> ...
> }
> size of test_table in bytes, 4285296
> length of test_table, 26532
> test_table, {
> [114.68.42.217] = 0,
> [4.48.74.147] = 0,
> [58.127.37.141] = 0,
> ...
> ...
> }
> size of test_table in bytes, 1526280
> length of test_table, 3
> test_table, {
> [x.x.229.229] = 0,
> [x.x.228.20] = 0,
> [x.x.229.5] = 0
> }
> size of test_table in bytes, 1526384
> length of test_table, 4
> test_table, {
> [x.x.201.77] = 0,
> [x.x.201.145] = 0,
> [x.x.201.49] = 0,
> [x.x.201.52] = 0
> }
> size of test_table in bytes, 1526072
> length of test_table, 1
> test_table, {
> [x.x.201.70] = 0
> }
> size of test_table in bytes, 1525968
> length of test_table, 0
> test_table, {
> 
> }
> size of test_table in bytes, 1526072
> length of test_table, 1
> test_table, {
> [x.x.229.229] = 0
> }
> size of test_table in bytes, 1526072
> length of test_table, 1
> test_table, {
> [x.x.229.229] = 0
> }


cheers
christoph