[Bro] Bro Memory Consumtion
William Jones
jones at tacc.utexas.edu
Fri Mar 19 12:58:26 PDT 2010
I think your memory usage is not too bad. Here is mine based on the output of top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19954 root 16 0 575m 512m 3264 R 36.1 6.4 2786:58 bro
19959 root 16 0 608m 546m 3208 R 26.3 6.8 2836:23 bro
19957 root 16 0 1876m 213m 3044 R 25.3 2.7 798:22.93 bro
19960 root 15 0 2269m 1.3g 2972 S 13.8 16.2 583:47.36 bro
19955 root 15 0 2242m 1.5g 2940 R 13.5 19.7 598:52.38 bro
19956 root 15 0 4381m 1.4g 3116 R 13.5 17.7 3207:07 bro
19958 root 15 0 2240m 94m 2892 R 5.9 1.2 1483:13 bro
20261 root 21 5 81400 3184 464 R 5.3 0.0 47:04.14 bro
20254 root 20 5 81400 3832 464 R 3.9 0.0 32:29.51 bro
20257 root 21 5 81400 2996 464 R 3.6 0.0 30:25.10 bro
20258 root 20 5 82584 4156 464 R 3.0 0.1 32:05.68 bro
20259 root 20 5 81664 3784 464 R 3.0 0.0 32:17.27 bro
20260 root 20 5 82584 4240 464 R 3.0 0.1 33:45.70 bro
20256 root 20 5 82664 3292 464 R 2.0 0.0 30:29.67 bro
14161 root 15 0 12740 1096 820 R 0.3 0.0 0:00.21 top
-----Original Message-----
From: bro-bounces at ICSI.Berkeley.EDU [mailto:bro-bounces at ICSI.Berkeley.EDU] On Behalf Of Powell, Scott
Sent: Friday, March 19, 2010 12:50 PM
To: Powell, Scott; Seth Hall
Cc: Justin Azoff; bro at ICSI.Berkeley.EDU
Subject: Re: [Bro] Bro Memory Consumtion
Seth and all,
My memory consumption is better but still growing and not shrinking. I've been examining the globals in the prof.log files for each of the various components (workers, manager, etc.) but am not sure what is causing so much memory to be allocated. Below is an example from one of my workers. There is ~3.6GB of memory allocated, total, but the globals are only 214MB. This is replicated across my three workers... plus the memory being used by the manager and proxy... so grand total I'm now up to ~12GB of allocated memory and it continues to grow.
Mar 19 13:42:20 ------------------------
Mar 19 13:42:20 Memory: total=3821576K total_adj=3765668K malloced: 3814486K
Mar 19 13:42:20 Run-time: user+sys=55905.4 user=53574.3 sys=2331.1 real=99872.4
Mar 19 13:42:20 Conns: total=12370755 current=4998/859 ext=0 mem=3372528K avg=3926.1 table=3430K connvals=2328K
Mar 19 13:42:20 ConnCompressor: pending=36 pending_in_mem=582 full_conns=-4895 pending+real=4175 mem=48K avg=1368.7/84.7
Mar 19 13:42:20 Conns: tcp=0/0 udp=844/1984 icmp=15/50
Mar 19 13:42:20 TCP-States: Inact. Syn. SA Part. Est. Fin. Rst.
Mar 19 13:42:20 TCP-States:Inact. 76 2 4
Mar 19 13:42:20 TCP-States:Syn.
Mar 19 13:42:20 TCP-States:SA
Mar 19 13:42:20 TCP-States:Part. 12 755 1 26
Mar 19 13:42:20 TCP-States:Est. 2412 98 7
Mar 19 13:42:20 TCP-States:Fin. 8 5 62 416 3
Mar 19 13:42:20 TCP-States:Rst. 95 12 90 54 1
Mar 19 13:42:20 Connections expired due to inactivity: 2012770
Mar 19 13:42:20 Total reassembler data: 236K
Mar 19 13:42:20 RuleMatcher: matchers=2 dfa_states=599 ncomputed=9765 mem=1309K avg_nfa_states=19
Mar 19 13:42:20 Timers: current=12852 max=19240 mem=1004K lag=0.00s
Mar 19 13:42:20 ConnectionDeleteTimer = 590
Mar 19 13:42:20 ConnectionInactivityTimer = 6874
Mar 19 13:42:20 DNSExpireTimer = 385
Mar 19 13:42:20 NetworkTimer = 1
Mar 19 13:42:20 NTPExpireTimer = 60
Mar 19 13:42:20 RotateTimer = 35
Mar 19 13:42:20 ScheduleTimer = 840
Mar 19 13:42:20 TableValTimer = 79
Mar 19 13:42:20 TCPConnectionAttemptTimer = 255
Mar 19 13:42:20 TCPConnectionExpireTimer = 3733
Mar 19 13:42:20 Global_sizes > 100k: 0K
Mar 19 13:42:20 SSH::did_ssh_version = 24K (109/109 entries)
Mar 19 13:42:20 Login::login_sessions = 122K (140/140 entries)
Mar 19 13:42:20 SMTP::smtp_sessions = 973K (17/17 entries)
Mar 19 13:42:20 KnownServices::established_conns = 191K (386/386 entries)
Mar 19 13:42:20 ssl_cipher_desc = 30K (106/106 entries)
Mar 19 13:42:20 dpd_analyzer_ports = 128K (35/700 entries)
Mar 19 13:42:20 Scan::rops_idx = 39K (171/171 entries)
Mar 19 13:42:20 notice_tags = 262K (690/690 entries)
Mar 19 13:42:20 KnownHosts::known_hosts = 1861K (14160/14160 entries)
Mar 19 13:42:20 Login::output_trouble = 399K
Mar 19 13:42:20 DNS::distinct_PTR_requests = 481K (648/648 entries)
Mar 19 13:42:20 Scan::distinct_ports = 5880K (5376/20084 entries)
Mar 19 13:42:20 HTTP::http_sessions = 9018K (1697/1697 entries)
Mar 19 13:42:20 ssl_connections = 2436K (905/905 entries)
Mar 19 13:42:20 ftp_cmd_reply_code = 40K (273/273 entries)
Mar 19 13:42:20 Weird::weird_ignore = 99K (94/188 entries)
Mar 19 13:42:20 DNS::distinct_answered_PTR_requests = 45K (145/145 entries)
Mar 19 13:42:20 SMTP::reject_counter = 5115K (9475/9475 entries)
Mar 19 13:42:20 Scan::distinct_backscatter_peers = 269K (126/724 entries)
Mar 19 13:42:20 DetectProtocolHTTP::conns = 438K (470/940 entries)
Mar 19 13:42:20 HTTP::sql_injection_regex = 603K
Mar 19 13:42:20 Scan::accounts_tried = 94K (96/222 entries)
Mar 19 13:42:20 Portmapper::rpc_programs = 35K (129/129 entries)
Mar 19 13:42:20 HTTP::known_user_agents = 10475K (8027/29020 entries)
Mar 19 13:42:20 Scan::possible_scan_sources = 14K (106/106 entries)
Mar 19 13:42:20 IRC::active_channels = 334K (47/47 entries)
Mar 19 13:42:20 ssl_sessionIDs = 117981K (27276/27276 entries)
Mar 19 13:42:20 FTP::hot_files = 112K
Mar 19 13:42:20 Scan::pre_distinct_peers = 31560K (35230/72640 entries)
Mar 19 13:42:20 HTTP::sensitive_URIs = 519K
Mar 19 13:42:20 DetectProtocolHTTP::protocols = 278K (7/7 entries)
Mar 19 13:42:20 Scan::distinct_low_ports = 89K (98/196 entries)
Mar 19 13:42:20 IRC::active_users = 525K (96/96 entries)
Mar 19 13:42:20 Scan::scan_triples = 7386K (106/17547 entries)
Mar 19 13:42:20 Software::host_software = 9502K (5079/10272 entries)
Mar 19 13:42:20 DNS::dns_sessions = 1011K (629/629 entries)
Mar 19 13:42:20 Scan::distinct_peers = 4584K (571/30458 entries)
Mar 19 13:42:20 HTTP::suspicious_http_posts = 733K
Mar 19 13:42:20 KnownServices::known_services = 42K (261/261 entries)
Mar 19 13:42:20 Login::input_trouble = 108K
Mar 19 13:42:20 Weird::weird_action = 39K (170/170 entries)
Mar 19 13:42:20 HTTP::conn_info = 3007K (759/759 entries)
Mar 19 13:42:20 Global_sizes total: 219225K
Mar 19 13:42:20 Total number of table entries: 115411/243104
Mar 19 13:42:35 ------------------------
PID 5562 (manager): 74112K 72.375M
PID 5932 (manager): 201492K 196.77M
PID 5950 (manager): 96240K 93.9844M
PID 5962 (proxy-1): 74112K 72.375M
PID 5974 (proxy-1): 144396K 141.012M
PID 5975 (proxy-1): 97824K 95.5312M
PID 6002 (worker-1): 74112K 72.375M
PID 6038 (worker-1): 3608784K 3524.2M
PID 6042 (worker-1): 94884K 92.6602M
PID 5999 (worker-2): 74112K 72.375M
PID 6036 (worker-2): 3966504K 3873.54M
PID 6040 (worker-2): 94888K 92.6641M
PID 6001 (worker-3): 74112K 72.375M
PID 6037 (worker-3): 3930396K 3838.28M
PID 6041 (worker-3): 93916K 91.7148M
Total: 12.1116G
Any ideas?
Thanks,
Scott
-----Original Message-----
From: bro-bounces at ICSI.Berkeley.EDU [mailto:bro-bounces at ICSI.Berkeley.EDU] On Behalf Of Powell, Scott
Sent: Thursday, March 18, 2010 1:26 PM
To: Seth Hall
Cc: Justin Azoff; bro at ICSI.Berkeley.EDU
Subject: Re: [Bro] Bro Memory Consumtion
Seth,
Thanks. I'm now running without the DNS scripts and have profiling enabled. I will see how it goes. Right now Bro is using about 4.5GB between the manager, proxy and my three workers (all running on the same system w/click splitting up the tap). I was restarting each day at 1am but I have commented out the cron. I'll check it in the morning and see if things are cleaning up after themselves.
Thanks,
Scott
-----Original Message-----
From: Seth Hall [mailto:hall.692 at osu.edu]
Sent: Thursday, March 18, 2010 9:51 AM
To: Powell, Scott
Cc: Justin Azoff; bro at ICSI.Berkeley.EDU
Subject: Re: [Bro] Bro Memory Consumtion
On Mar 18, 2010, at 9:25 AM, Powell, Scott wrote:
> I synced my scripts up with the latest and greatest from Seth's
> repository but am still seeing Bro consume all 16gb of memory after
> only an hour or two. When time permits I will try to debug further
> to see if I can narrow it down to a particular script/policy.
I just moved both of the dns scripts into the testing/ directory to
clear up any confusion about their stability. :) When I get time and
make them better with memory I'll move them back to the main directory.
.Seth
---
Seth Hall
Network Security - Office of the CIO
The Ohio State University
Phone: 614-292-9721
_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
More information about the Bro
mailing list