[Bro] BRO Logger crashing due to large DNS log files
Ron McClellan
Ron_McClellan at ao.uscourts.gov
Wed Aug 22 08:40:54 PDT 2018
Sorry, forgot to send that, I did re-enable the conn.log.
Ron
#################################################################
# Checking if many recent connections have a SAD or had history #
#################################################################
error: 24.30%, 52 out of 214 connections are half duplex
################################################
# Checking for recent capture_loss.log entries #
################################################
Capture loss stats:
worker=worker-1-1 loss_count=169 noloss_count=5 min_loss=0.0 max_loss=49.699158 overall_loss=0.711248084022
worker=worker-1-10 loss_count=165 noloss_count=9 min_loss=0.0 max_loss=31.734317 overall_loss=0.708971591962
worker=worker-1-11 loss_count=163 noloss_count=7 min_loss=0.0 max_loss=49.215598 overall_loss=0.733236081253
worker=worker-1-12 loss_count=172 noloss_count=7 min_loss=0.0 max_loss=19.54023 overall_loss=0.698003997145
worker=worker-1-13 loss_count=170 noloss_count=3 min_loss=0.0 max_loss=59.151943 overall_loss=0.708047294873
worker=worker-1-14 loss_count=166 noloss_count=4 min_loss=0.0 max_loss=31.102362 overall_loss=0.729493755422
worker=worker-1-15 loss_count=175 noloss_count=5 min_loss=0.0 max_loss=23.337269 overall_loss=0.696221193462
worker=worker-1-16 loss_count=171 noloss_count=3 min_loss=0.0 max_loss=21.665887 overall_loss=0.730258043624
worker=worker-1-17 loss_count=166 noloss_count=2 min_loss=0.0 max_loss=24.537037 overall_loss=0.730114941561
worker=worker-1-18 loss_count=173 noloss_count=5 min_loss=0.0 max_loss=35.149385 overall_loss=0.707049677317
worker=worker-1-19 loss_count=170 noloss_count=2 min_loss=0.0 max_loss=29.000411 overall_loss=0.715538683512
worker=worker-1-2 loss_count=167 noloss_count=3 min_loss=0.0 max_loss=25.104778 overall_loss=0.725878156063
worker=worker-1-20 loss_count=157 noloss_count=11 min_loss=0.0 max_loss=27.755102 overall_loss=0.732279462789
worker=worker-1-21 loss_count=175 noloss_count=3 min_loss=0.0 max_loss=21.428571 overall_loss=0.709650475913
worker=worker-1-22 loss_count=166 noloss_count=4 min_loss=0.0 max_loss=47.108131 overall_loss=0.723601858469
worker=worker-1-23 loss_count=163 noloss_count=5 min_loss=0.0 max_loss=26.769231 overall_loss=0.737703465832
worker=worker-1-24 loss_count=171 noloss_count=5 min_loss=0.0 max_loss=29.166667 overall_loss=0.70277340392
worker=worker-1-25 loss_count=167 noloss_count=5 min_loss=0.0 max_loss=43.062904 overall_loss=0.719155267119
worker=worker-1-26 loss_count=166 noloss_count=2 min_loss=0.0 max_loss=37.043478 overall_loss=0.733480109044
worker=worker-1-27 loss_count=172 noloss_count=7 min_loss=0.0 max_loss=21.570681 overall_loss=0.702686952111
worker=worker-1-28 loss_count=168 noloss_count=4 min_loss=0.0 max_loss=63.5 overall_loss=0.715802336358
worker=worker-1-29 loss_count=163 noloss_count=5 min_loss=0.0 max_loss=21.200387 overall_loss=0.726079337034
worker=worker-1-3 loss_count=178 noloss_count=3 min_loss=0.0 max_loss=21.5 overall_loss=0.695671670577
worker=worker-1-30 loss_count=170 noloss_count=7 min_loss=0.0 max_loss=52.182163 overall_loss=0.711452182534
worker=worker-1-31 loss_count=101 noloss_count=0 min_loss=0.249393 max_loss=2.802944 overall_loss=0.734132084543
worker=worker-1-32 loss_count=100 noloss_count=0 min_loss=0.232239 max_loss=2.823435 overall_loss=0.732152381242
worker=worker-1-33 loss_count=111 noloss_count=0 min_loss=0.245892 max_loss=2.823399 overall_loss=0.715639440645
worker=worker-1-34 loss_count=103 noloss_count=0 min_loss=0.222024 max_loss=2.813374 overall_loss=0.711321116497
worker=worker-1-35 loss_count=100 noloss_count=0 min_loss=0.219738 max_loss=2.830536 overall_loss=0.728313426089
worker=worker-1-36 loss_count=111 noloss_count=0 min_loss=0.242244 max_loss=2.810973 overall_loss=0.711242220111
worker=worker-1-37 loss_count=104 noloss_count=0 min_loss=0.219657 max_loss=2.804831 overall_loss=0.710176458823
worker=worker-1-38 loss_count=100 noloss_count=0 min_loss=0.219585 max_loss=2.808788 overall_loss=0.724778339046
worker=worker-1-39 loss_count=111 noloss_count=0 min_loss=0.219585 max_loss=2.803075 overall_loss=0.695610695301
worker=worker-1-4 loss_count=170 noloss_count=4 min_loss=0.0 max_loss=44.75899 overall_loss=0.710287169691
worker=worker-1-40 loss_count=104 noloss_count=0 min_loss=0.219585 max_loss=2.8021 overall_loss=0.70998112771
worker=worker-1-41 loss_count=100 noloss_count=0 min_loss=0.219585 max_loss=2.802619 overall_loss=0.72342640507
worker=worker-1-42 loss_count=110 noloss_count=0 min_loss=0.219442 max_loss=2.809179 overall_loss=0.694948046234
worker=worker-1-43 loss_count=104 noloss_count=0 min_loss=0.219442 max_loss=2.799245 overall_loss=0.709799229819
worker=worker-1-44 loss_count=100 noloss_count=0 min_loss=0.219513 max_loss=2.799373 overall_loss=0.721964444083
worker=worker-1-45 loss_count=111 noloss_count=0 min_loss=0.219441 max_loss=2.80574 overall_loss=0.695340745288
worker=worker-1-46 loss_count=104 noloss_count=0 min_loss=0.219585 max_loss=2.806646 overall_loss=0.710300132624
error: worker=worker-1-47 loss_count=42 noloss_count=0 min_loss=0.252768 max_loss=2.803858 overall_loss=1.00735434535
worker=worker-1-48 loss_count=55 noloss_count=0 min_loss=0.252767 max_loss=2.803207 overall_loss=0.874731142836
error: worker=worker-1-49 loss_count=20 noloss_count=0 min_loss=0.827054 max_loss=2.806194 overall_loss=1.3206719637
worker=worker-1-5 loss_count=164 noloss_count=6 min_loss=0.0 max_loss=28.184893 overall_loss=0.721071748929
error: worker=worker-1-50 loss_count=20 noloss_count=0 min_loss=0.828338 max_loss=2.807944 overall_loss=1.32179609476
worker=worker-1-6 loss_count=178 noloss_count=3 min_loss=0.0 max_loss=31.229236 overall_loss=0.69677119794
worker=worker-1-7 loss_count=170 noloss_count=4 min_loss=0.0 max_loss=23.232323 overall_loss=0.706048088686
worker=worker-1-8 loss_count=169 noloss_count=1 min_loss=0.0 max_loss=28.685259 overall_loss=0.720893457905
worker=worker-1-9 loss_count=174 noloss_count=7 min_loss=0.0 max_loss=21.649485 overall_loss=0.693994393064
################################################################
# Checking what percentage of recent tcp connections show loss #
################################################################
error: 13.08%, 28 out of 214 connections have capture loss
###################################################################
# Checking if connections are unevenly distributed across workers #
###################################################################
error: No node names in conn log. Install add-node-names package to add a corresponding field.
###############################################################################################################################
# Checking if anything is in the deprecated local-logger.bro, local-manager.bro, local-proxy.bro, or local-worker.bro scripts #
###############################################################################################################################
Nothing found
######################################################################
# Checking if any recent connections have been logged multiple times #
######################################################################
ok, only 0.00%, 0 out of 19 connections appear to be duplicate
############################################################################
# Checking what percentage of recent tcp connections are remote to remote. #
############################################################################
0.00%, 0 out of 100000 connections are remote to remote
###############################################################################
# Checking if bro is linked against a custom malloc like tcmalloc or jemalloc #
###############################################################################
error: configured to use a custom malloc=False
##################################
# Checking pf_ring configuration #
##################################
configured to use pf_ring=False pcap=False plugin=False
############################################
# Checking for recent reporter.log entries #
############################################
error: Found 66 reporter log files in the past 7 days
Recent reporter.log messages:
Reporter::INFO processing suspended
Reporter::INFO processing suspended
305 duplicate messages suppressed
Reporter::INFO processing continued
Reporter::INFO processing continued
344 duplicate messages suppressed
1534945569.655686 Reporter::INFO received termination signal
1534945569.657754 Reporter::INFO received termination signal
174 duplicate messages suppressed
1534945569.655686 Reporter::INFO 59878 packets received on interface ens1f0, 0 dropped
1534945569.657754 Reporter::INFO 56518 packets received on interface ens1f0, 0 dropped
1534945569.660757 Reporter::INFO 56724 packets received on interface ens1f0, 0 dropped
1534945569.658804 Reporter::INFO 54919 packets received on interface ens1f0, 0 dropped
1534945569.663107 Reporter::INFO 55802 packets received on interface ens1f0, 0 dropped
1534945569.665709 Reporter::INFO 55495 packets received on interface ens1f0, 0 dropped
1534945569.664417 Reporter::INFO 56303 packets received on interface ens1f0, 0 dropped
1534945569.661807 Reporter::INFO 54850 packets received on interface ens1f0, 0 dropped
1534945569.652757 Reporter::INFO 56683 packets received on interface ens1f0, 0 dropped
1534945569.666680 Reporter::INFO 728568 packets received on interface ens1f0, 0 dropped
1534945569.668403 Reporter::INFO 54410 packets received on interface ens1f0, 0 dropped
1534945569.659922 Reporter::INFO 66057 packets received on interface ens1f0, 0 dropped
1534945569.670068 Reporter::INFO 96452 packets received on interface ens1f0, 0 dropped
1534945569.654122 Reporter::INFO 60502 packets received on interface ens1f0, 0 dropped
1534945569.674937 Reporter::INFO 534148 packets received on interface ens1f0, 0 dropped
1534945569.676096 Reporter::INFO 59423 packets received on interface ens1f0, 0 dropped
1534945569.671680 Reporter::INFO 54199 packets received on interface ens1f0, 0 dropped
1534945569.675688 Reporter::INFO 55103 packets received on interface ens1f0, 0 dropped
1534945569.672645 Reporter::INFO 55276 packets received on interface ens1f0, 0 dropped
1534945569.681012 Reporter::INFO 59191 packets received on interface ens1f0, 0 dropped
1534945569.685158 Reporter::INFO 526830 packets received on interface ens1f0, 0 dropped
1534945569.682198 Reporter::INFO 56117 packets received on interface ens1f0, 0 dropped
1534945569.686343 Reporter::INFO 280179 packets received on interface ens1f0, 0 dropped
1534945569.685319 Reporter::INFO 53622 packets received on interface ens1f0, 0 dropped
1534945569.690807 Reporter::INFO 55072 packets received on interface ens1f0, 0 dropped
1534945569.679809 Reporter::INFO 425150 packets received on interface ens1f0, 0 dropped
1534945569.680279 Reporter::INFO 57843 packets received on interface ens1f0, 0 dropped
1534945569.692148 Reporter::INFO 54710 packets received on interface ens1f0, 0 dropped
1534945569.677281 Reporter::INFO 55582 packets received on interface ens1f0, 0 dropped
1534945569.689345 Reporter::INFO 67500 packets received on interface ens1f0, 0 dropped
1534944921.064766 Reporter::INFO 7476939 packets received on interface ens1f0, 0 dropped
1534944921.066590 Reporter::INFO 5979272 packets received on interface ens1f0, 0 dropped
1534944921.069660 Reporter::INFO 11350634 packets received on interface ens1f0, 0 dropped
1534944921.068621 Reporter::INFO 7846634 packets received on interface ens1f0, 0 dropped
1534944921.070665 Reporter::INFO 7716706 packets received on interface ens1f0, 0 dropped
1534944921.072682 Reporter::INFO 8075699 packets received on interface ens1f0, 0 dropped
1534944921.073648 Reporter::INFO 7254555 packets received on interface ens1f0, 0 dropped
1534944921.071549 Reporter::INFO 8783077 packets received on interface ens1f0, 0 dropped
1534944921.076533 Reporter::INFO 7179347 packets received on interface ens1f0, 0 dropped
1534944921.075584 Reporter::INFO 7727934 packets received on interface ens1f0, 0 dropped
1534944921.074546 Reporter::INFO 6784747 packets received on interface ens1f0, 0 dropped
1534944921.078555 Reporter::INFO 6505658 packets received on interface ens1f0, 0 dropped
1534944921.079501 Reporter::INFO 11272110 packets received on interface ens1f0, 0 dropped
1534944921.077461 Reporter::INFO 11702900 packets received on interface ens1f0, 0 dropped
1534944921.083506 Reporter::INFO 9770050 packets received on interface ens1f0, 0 dropped
1534944921.084694 Reporter::INFO 5377432 packets received on interface ens1f0, 0 dropped
1534944921.080490 Reporter::INFO 7853539 packets received on interface ens1f0, 0 dropped
1534944921.089195 Reporter::INFO 7411832 packets received on interface ens1f0, 0 dropped
1534944921.088097 Reporter::INFO 20093065 packets received on interface ens1f0, 0 dropped
1534944921.081517 Reporter::INFO 10139208 packets received on interface ens1f0, 0 dropped
1534944921.082494 Reporter::INFO 9928228 packets received on interface ens1f0, 0 dropped
1534944921.085820 Reporter::INFO 9309141 packets received on interface ens1f0, 0 dropped
1534944921.092678 Reporter::INFO 7987870 packets received on interface ens1f0, 0 dropped
1534944921.093619 Reporter::INFO 9510548 packets received on interface ens1f0, 0 dropped
1534944921.095941 Reporter::INFO 7619163 packets received on interface ens1f0, 0 dropped
1534944921.071034 Reporter::INFO 7962640 packets received on interface ens1f0, 2888 dropped
1534944921.099008 Reporter::INFO 7523835 packets received on interface ens1f0, 0 dropped
1534944921.101288 Reporter::INFO 8860676 packets received on interface ens1f0, 0 dropped
1534944921.102270 Reporter::INFO 7432686 packets received on interface ens1f0, 0 dropped
1534944921.099397 Reporter::INFO 8637163 packets received on interface ens1f0, 0 dropped
/usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro, line 54 1534945591.937813 Reporter::WARNING Your interface is likely receiving invalid TCP checksums, most likely from NIC checksum offloading. By default, packets with invalid checksums are discarded by Bro unless using the -C command-line option or toggling the 'ignore_checksums' variable. Alternatively, disable checksum offloading by the network adapter to ensure Bro analyzes the actual checksums that are transmitted.
/usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro, line 54 1534891136.995923 Reporter::WARNING Your interface is likely receiving invalid TCP checksums, most likely from NIC checksum offloading. By default, packets with invalid checksums are discarded by Bro unless using the -C command-line option or toggling the 'ignore_checksums' variable. Alternatively, disable checksum offloading by the network adapter to ensure Bro analyzes the actual checksums that are transmitted.
5 duplicate messages suppressed
1534891114.741981 Reporter::INFO 372014 packets received on interface ens1f0, 0 dropped
1534891114.743404 Reporter::INFO 1220128 packets received on interface ens1f0, 0 dropped
1534891114.744481 Reporter::INFO 661511 packets received on interface ens1f0, 0 dropped
1534891114.745450 Reporter::INFO 576353 packets received on interface ens1f0, 0 dropped
1534891114.746610 Reporter::INFO 186013 packets received on interface ens1f0, 0 dropped
1534891114.747555 Reporter::INFO 1719908 packets received on interface ens1f0, 0 dropped
1534891114.749554 Reporter::INFO 182315 packets received on interface ens1f0, 0 dropped
1534891114.750730 Reporter::INFO 9263131 packets received on interface ens1f0, 0 dropped
1534891114.754252 Reporter::INFO 920965 packets received on interface ens1f0, 0 dropped
1534891114.740742 Reporter::INFO 184512 packets received on interface ens1f0, 0 dropped
1534891114.755328 Reporter::INFO 191456 packets received on interface ens1f0, 0 dropped
1534891114.752679 Reporter::INFO 556688 packets received on interface ens1f0, 0 dropped
1534891114.756511 Reporter::INFO 775373 packets received on interface ens1f0, 0 dropped
1534891114.739297 Reporter::INFO 898836 packets received on interface ens1f0, 0 dropped
1534891114.757522 Reporter::INFO 185947 packets received on interface ens1f0, 0 dropped
1534891114.759550 Reporter::INFO 196218 packets received on interface ens1f0, 0 dropped
1534891114.759936 Reporter::INFO 194993 packets received on interface ens1f0, 0 dropped
1534891114.762748 Reporter::INFO 834002 packets received on interface ens1f0, 0 dropped
1534891114.764149 Reporter::INFO 193840 packets received on interface ens1f0, 0 dropped
1534891114.762527 Reporter::INFO 186095 packets received on interface ens1f0, 0 dropped
1534891114.765131 Reporter::INFO 259919 packets received on interface ens1f0, 0 dropped
1534891114.748692 Reporter::INFO 1396336 packets received on interface ens1f0, 0 dropped
1534891114.751699 Reporter::INFO 179743 packets received on interface ens1f0, 0 dropped
1534891114.767549 Reporter::INFO 187725 packets received on interface ens1f0, 0 dropped
1534891114.767200 Reporter::INFO 811731 packets received on interface ens1f0, 0 dropped
1534891114.771534 Reporter::INFO 187778 packets received on interface ens1f0, 0 dropped
1534891114.769285 Reporter::INFO 207081 packets received on interface ens1f0, 0 dropped
1534891114.768756 Reporter::INFO 225592 packets received on interface ens1f0, 0 dropped
1534891114.770325 Reporter::INFO 190033 packets received on interface ens1f0, 0 dropped
1534891114.772342 Reporter::INFO 182314 packets received on interface ens1f0, 0 dropped
1534883904.208832 Reporter::INFO 80395 packets received on interface ens1f0, 213 dropped
1534883904.212230 Reporter::INFO 80261 packets received on interface ens1f0, 248 dropped
1534883904.210355 Reporter::INFO 80721 packets received on interface ens1f0, 205 dropped
1534883904.205811 Reporter::INFO 80689 packets received on interface ens1f0, 211 dropped
1534883904.202076 Reporter::INFO 79672 packets received on interface ens1f0, 317 dropped
1534883904.204229 Reporter::INFO 79679 packets received on interface ens1f0, 317 dropped
1534883904.187896 Reporter::INFO 79594 packets received on interface ens1f0, 316 dropped
1534883904.197151 Reporter::INFO 79645 packets received on interface ens1f0, 315 dropped
1534883904.193226 Reporter::INFO 79615 packets received on interface ens1f0, 315 dropped
1534883904.195856 Reporter::INFO 79643 packets received on interface ens1f0, 317 dropped
1534883904.186647 Reporter::INFO 79592 packets received on interface ens1f0, 314 dropped
1534883904.200680 Reporter::INFO 79667 packets received on interface ens1f0, 316 dropped
1534883904.192134 Reporter::INFO 79550 packets received on interface ens1f0, 311 dropped
1534883904.189401 Reporter::INFO 79534 packets received on interface ens1f0, 315 dropped
1534883904.194799 Reporter::INFO 79564 packets received on interface ens1f0, 304 dropped
1534883904.192234 Reporter::INFO 79542 packets received on interface ens1f0, 298 dropped
1534886979.721378 Reporter::INFO 1072583 packets received on interface ens1f0, 141 dropped
1534886979.732950 Reporter::INFO 1070920 packets received on interface ens1f0, 275 dropped
1534886979.724258 Reporter::INFO 1072912 packets received on interface ens1f0, 0 dropped
1534886979.729367 Reporter::INFO 1070892 packets received on interface ens1f0, 274 dropped
1534886979.734314 Reporter::INFO 1072735 packets received on interface ens1f0, 0 dropped
1534886979.725162 Reporter::INFO 1072659 packets received on interface ens1f0, 0 dropped
1534886979.727819 Reporter::INFO 1072688 packets received on interface ens1f0, 0 dropped
1534886979.730567 Reporter::INFO 1072646 packets received on interface ens1f0, 127 dropped
1534886979.722836 Reporter::INFO 1072663 packets received on interface ens1f0, 0 dropped
1534886979.731947 Reporter::INFO 1072706 packets received on interface ens1f0, 0 dropped
1534886979.736491 Reporter::INFO 1072677 packets received on interface ens1f0, 141 dropped
1534886979.735473 Reporter::INFO 1070923 packets received on interface ens1f0, 283 dropped
1534886979.740571 Reporter::INFO 1070961 packets received on interface ens1f0, 1645 dropped
1534886979.738937 Reporter::INFO 1070963 packets received on interface ens1f0, 1634 dropped
1534886979.745314 Reporter::INFO 1070974 packets received on interface ens1f0, 1658 dropped
1534886979.746646 Reporter::INFO 1070971 packets received on interface ens1f0, 1645 dropped
1534886979.741628 Reporter::INFO 1070914 packets received on interface ens1f0, 1631 dropped
1534886979.737836 Reporter::INFO 1070932 packets received on interface ens1f0, 1644 dropped
1534886979.747760 Reporter::INFO 1070998 packets received on interface ens1f0, 1645 dropped
1534886979.743048 Reporter::INFO 1070961 packets received on interface ens1f0, 1646 dropped
1534886979.752948 Reporter::INFO 1071049 packets received on interface ens1f0, 1645 dropped
1534886979.756786 Reporter::INFO 1071055 packets received on interface ens1f0, 1645 dropped
1534886979.755328 Reporter::INFO 1071055 packets received on interface ens1f0, 1645 dropped
1534886979.748970 Reporter::INFO 1071019 packets received on interface ens1f0, 1645 dropped
1534886979.751311 Reporter::INFO 1071005 packets received on interface ens1f0, 1645 dropped
1534886979.744325 Reporter::INFO 1070921 packets received on interface ens1f0, 1636 dropped
1534886979.758477 Reporter::INFO 1071064 packets received on interface ens1f0, 1656 dropped
1534886979.750125 Reporter::INFO 1071021 packets received on interface ens1f0, 1645 dropped
1534886979.761657 Reporter::INFO 1071083 packets received on interface ens1f0, 1645 dropped
1534886979.759686 Reporter::INFO 1071089 packets received on interface ens1f0, 1645 dropped
1534885764.945067 Reporter::INFO 19126359 packets received on interface ens1f0, 0 dropped
1534885764.945531 Reporter::INFO 19126359 packets received on interface ens1f0, 0 dropped
1534885764.946470 Reporter::INFO 19126368 packets received on interface ens1f0, 0 dropped
1534885764.951497 Reporter::INFO 19126406 packets received on interface ens1f0, 0 dropped
1534885764.957197 Reporter::INFO 19126436 packets received on interface ens1f0, 0 dropped
1534885764.940357 Reporter::INFO 19126342 packets received on interface ens1f0, 0 dropped
1534885764.949431 Reporter::INFO 19126391 packets received on interface ens1f0, 0 dropped
1534885764.942220 Reporter::INFO 19126355 packets received on interface ens1f0, 0 dropped
1534885764.952726 Reporter::INFO 19126408 packets received on interface ens1f0, 0 dropped
1534885764.959421 Reporter::INFO 19126450 packets received on interface ens1f0, 0 dropped
1534885764.954249 Reporter::INFO 19126417 packets received on interface ens1f0, 0 dropped
1534885764.950288 Reporter::INFO 19126397 packets received on interface ens1f0, 0 dropped
1534885764.955658 Reporter::INFO 19126430 packets received on interface ens1f0, 0 dropped
1534885764.957783 Reporter::INFO 19126441 packets received on interface ens1f0, 0 dropped
1534885764.963722 Reporter::INFO 19126483 packets received on interface ens1f0, 0 dropped
1534885764.962441 Reporter::INFO 19126469 packets received on interface ens1f0, 0 dropped
1534896852.839809 Reporter::INFO 1030981 packets received on interface ens1f0, 0 dropped
1534896852.838278 Reporter::INFO 392308 packets received on interface ens1f0, 0 dropped
1534896852.841137 Reporter::INFO 818619 packets received on interface ens1f0, 0 dropped
1534896852.844479 Reporter::INFO 1067255 packets received on interface ens1f0, 0 dropped
1534896852.842345 Reporter::INFO 1029386 packets received on interface ens1f0, 0 dropped
1534896852.843326 Reporter::INFO 841578 packets received on interface ens1f0, 0 dropped
1534896852.848442 Reporter::INFO 1045593 packets received on interface ens1f0, 0 dropped
1534896852.845493 Reporter::INFO 551282 packets received on interface ens1f0, 0 dropped
1534896852.846489 Reporter::INFO 2077035 packets received on interface ens1f0, 0 dropped
1534896852.847490 Reporter::INFO 1160096 packets received on interface ens1f0, 0 dropped
1534896852.849339 Reporter::INFO 995455 packets received on interface ens1f0, 0 dropped
1534896852.853370 Reporter::INFO 1027908 packets received on interface ens1f0, 0 dropped
====================================================================
#################################################################
# Checking if many recent connections have a SAD or had history #
#################################################################
error: 75.53%, 14289 out of 18918 connections are half duplex
################################################
# Checking for recent capture_loss.log entries #
################################################
Capture loss stats:
worker=bro loss_count=0 noloss_count=168 min_loss=0.0 max_loss=0.0 overall_loss=0.0
error: worker=worker-1-1 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=90.58044 overall_loss=30.0099164747
error: worker=worker-1-10 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=66.577529 overall_loss=26.2442908547
error: worker=worker-1-11 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=67.87389 overall_loss=26.7675967606
error: worker=worker-1-12 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=77.820137 overall_loss=27.4078064757
error: worker=worker-1-13 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=67.574163 overall_loss=27.6988371111
error: worker=worker-1-14 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=63.196486 overall_loss=27.1931529853
error: worker=worker-1-15 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=69.743213 overall_loss=32.3055577642
error: worker=worker-1-16 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=65.566384 overall_loss=26.8274826352
error: worker=worker-1-17 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=66.476362 overall_loss=26.5423912274
error: worker=worker-1-18 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=64.410006 overall_loss=24.3477958539
error: worker=worker-1-19 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=63.185604 overall_loss=27.4475582221
error: worker=worker-1-2 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=69.678688 overall_loss=28.7640239525
error: worker=worker-1-20 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=63.994922 overall_loss=23.2474631419
error: worker=worker-1-21 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=65.539711 overall_loss=27.260502257
error: worker=worker-1-22 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=69.499069 overall_loss=30.2300032238
error: worker=worker-1-23 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=64.149083 overall_loss=31.1798858888
error: worker=worker-1-24 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=66.324143 overall_loss=22.5580314778
error: worker=worker-1-25 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=69.687833 overall_loss=29.667065343
error: worker=worker-1-26 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=65.863473 overall_loss=31.1213730204
error: worker=worker-1-27 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=65.497259 overall_loss=26.6064033512
error: worker=worker-1-28 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=64.144564 overall_loss=30.1043332527
error: worker=worker-1-29 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=64.903986 overall_loss=29.6894566048
error: worker=worker-1-3 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=68.130508 overall_loss=25.3573669197
error: worker=worker-1-30 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=72.621044 overall_loss=24.3457544317
worker=worker-1-31 loss_count=0 noloss_count=1 min_loss=0.0 max_loss=0.0 overall_loss=0.0
worker=worker-1-32 loss_count=0 noloss_count=1 min_loss=0.0 max_loss=0.0 overall_loss=0.0
worker=worker-1-33 loss_count=0 noloss_count=1 min_loss=0.0 max_loss=0.0 overall_loss=0.0
worker=worker-1-34 loss_count=0 noloss_count=1 min_loss=0.0 max_loss=0.0 overall_loss=0.0
worker=worker-1-35 loss_count=0 noloss_count=1 min_loss=0.0 max_loss=0.0 overall_loss=0.0
worker=worker-1-36 loss_count=0 noloss_count=1 min_loss=0.0 max_loss=0.0 overall_loss=0.0
error: worker=worker-1-4 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=67.538442 overall_loss=29.82586108
error: worker=worker-1-5 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=71.771296 overall_loss=31.4585270245
error: worker=worker-1-6 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=71.430447 overall_loss=27.4405648085
error: worker=worker-1-7 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=66.640978 overall_loss=25.8919190418
error: worker=worker-1-8 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=81.943911 overall_loss=34.0497839132
error: worker=worker-1-9 loss_count=69 noloss_count=1 min_loss=0.0 max_loss=72.243329 overall_loss=31.5214084629
################################################################
# Checking what percentage of recent tcp connections show loss #
################################################################
error: 13.21%, 2499 out of 18918 connections have capture loss
###################################################################
# Checking if connections are unevenly distributed across workers #
###################################################################
error: No node names in conn log. Install add-node-names package to add a corresponding field.
###############################################################################################################################
# Checking if anything is in the deprecated local-logger.bro, local-manager.bro, local-proxy.bro, or local-worker.bro scripts #
###############################################################################################################################
Nothing found
######################################################################
# Checking if any recent connections have been logged multiple times #
######################################################################
ok, only 0.00%, 0 out of 694 connections appear to be duplicate
############################################################################
# Checking what percentage of recent tcp connections are remote to remote. #
############################################################################
error: 52.85%, 52853 out of 100000 connections are remote to remote
###############################################################################
# Checking if bro is linked against a custom malloc like tcmalloc or jemalloc #
###############################################################################
error: configured to use a custom malloc=False
##################################
# Checking pf_ring configuration #
##################################
configured to use pf_ring=False pcap=False plugin=False
############################################
# Checking for recent reporter.log entries #
############################################
error: Found 14 reporter log files in the past 7 days
Recent reporter.log messages:
/usr/local/bro/share/bro/policy/protocols/ssh/geo-data.bro, line 30 1534951863.435398 Reporter::ERROR Bro was not configured for GeoIP support (lookup_location(SSH::lookup_ip))
/usr/local/bro/share/bro/policy/protocols/ssh/geo-data.bro, line 30 1534945001.027149 Reporter::ERROR Bro was not configured for GeoIP support (lookup_location(SSH::lookup_ip))
84 duplicate messages suppressed
1534944972.753733 Reporter::INFO processing suspended
1534944972.748734 Reporter::INFO processing suspended
133 duplicate messages suppressed
1534944972.753733 Reporter::INFO processing continued
1534944972.748734 Reporter::INFO processing continued
154 duplicate messages suppressed
/usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro, line 54 1534944982.705403 Reporter::WARNING Your interface is likely receiving invalid TCP checksums, most likely from NIC checksum offloading. By default, packets with invalid checksums are discarded by Bro unless using the -C command-line option or toggling the 'ignore_checksums' variable. Alternatively, disable checksum offloading by the network adapter to ensure Bro analyzes the actual checksums that are transmitted.
/usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro, line 54 1534891137.279399 Reporter::WARNING Your interface is likely receiving invalid TCP checksums, most likely from NIC checksum offloading. By default, packets with invalid checksums are discarded by Bro unless using the -C command-line option or toggling the 'ignore_checksums' variable. Alternatively, disable checksum offloading by the network adapter to ensure Bro analyzes the actual checksums that are transmitted.
49 duplicate messages suppressed
1534944917.319061 Reporter::INFO received termination signal
1534944917.325143 Reporter::INFO received termination signal
62 duplicate messages suppressed
1534944917.319061 Reporter::INFO 59439914 packets received on interface ens1f0, 0 dropped
1534944917.325143 Reporter::INFO 45022136 packets received on interface ens1f0, 0 dropped
1534944917.321217 Reporter::INFO 51627314 packets received on interface ens1f0, 0 dropped
1534944917.322883 Reporter::INFO 53891270 packets received on interface ens1f0, 0 dropped
1534944917.324085 Reporter::INFO 88417370 packets received on interface ens1f0, 0 dropped
1534944917.319957 Reporter::INFO 54791897 packets received on interface ens1f0, 0 dropped
1534944917.327305 Reporter::INFO 43678446 packets received on interface ens1f0, 0 dropped
1534944917.330087 Reporter::INFO 59703116 packets received on interface ens1f0, 0 dropped
1534944917.328208 Reporter::INFO 45211345 packets received on interface ens1f0, 0 dropped
1534944917.331134 Reporter::INFO 65262767 packets received on interface ens1f0, 0 dropped
1534944917.334825 Reporter::INFO 55188270 packets received on interface ens1f0, 0 dropped
1534944917.329257 Reporter::INFO 52034434 packets received on interface ens1f0, 0 dropped
1534944917.332107 Reporter::INFO 49495024 packets received on interface ens1f0, 0 dropped
1534944917.333015 Reporter::INFO 49204592 packets received on interface ens1f0, 0 dropped
1534944917.336672 Reporter::INFO 50396962 packets received on interface ens1f0, 0 dropped
1534944917.333880 Reporter::INFO 54559564 packets received on interface ens1f0, 0 dropped
1534944917.338831 Reporter::INFO 55635174 packets received on interface ens1f0, 0 dropped
1534944917.341397 Reporter::INFO 58206831 packets received on interface ens1f0, 0 dropped
1534944917.344880 Reporter::INFO 56679728 packets received on interface ens1f0, 0 dropped
1534944917.343923 Reporter::INFO 50649583 packets received on interface ens1f0, 0 dropped
1534944917.345953 Reporter::INFO 69599231 packets received on interface ens1f0, 0 dropped
1534944917.340038 Reporter::INFO 52187914 packets received on interface ens1f0, 0 dropped
1534944917.342584 Reporter::INFO 54560034 packets received on interface ens1f0, 0 dropped
1534944917.346950 Reporter::INFO 43540603 packets received on interface ens1f0, 0 dropped
1534944917.351083 Reporter::INFO 94170430 packets received on interface ens1f0, 0 dropped
1534944917.326276 Reporter::INFO 61679606 packets received on interface ens1f0, 0 dropped
1534944917.348123 Reporter::INFO 52021287 packets received on interface ens1f0, 0 dropped
1534944917.350083 Reporter::INFO 59524072 packets received on interface ens1f0, 0 dropped
1534944917.349146 Reporter::INFO 53404009 packets received on interface ens1f0, 0 dropped
1534944917.337775 Reporter::INFO 53757378 packets received on interface ens1f0, 0 dropped
1534891115.907765 Reporter::INFO 5219842 packets received on interface ens1f0, 0 dropped
1534891115.910010 Reporter::INFO 8455238 packets received on interface ens1f0, 0 dropped
1534891115.912626 Reporter::INFO 5228397 packets received on interface ens1f0, 0 dropped
1534891115.908858 Reporter::INFO 5747798 packets received on interface ens1f0, 0 dropped
1534891115.911476 Reporter::INFO 5697178 packets received on interface ens1f0, 0 dropped
1534891115.915011 Reporter::INFO 4743597 packets received on interface ens1f0, 0 dropped
1534891115.918402 Reporter::INFO 6230492 packets received on interface ens1f0, 0 dropped
1534891115.913823 Reporter::INFO 7316772 packets received on interface ens1f0, 0 dropped
1534891115.920544 Reporter::INFO 8210898 packets received on interface ens1f0, 0 dropped
1534891115.919012 Reporter::INFO 5552873 packets received on interface ens1f0, 0 dropped
1534891115.922581 Reporter::INFO 4081197 packets received on interface ens1f0, 0 dropped
1534891115.925757 Reporter::INFO 5280114 packets received on interface ens1f0, 0 dropped
1534891115.923134 Reporter::INFO 5538765 packets received on interface ens1f0, 0 dropped
1534891115.927881 Reporter::INFO 5320840 packets received on interface ens1f0, 0 dropped
1534891115.934273 Reporter::INFO 5085653 packets received on interface ens1f0, 0 dropped
1534891115.928977 Reporter::INFO 9046510 packets received on interface ens1f0, 0 dropped
1534891115.921576 Reporter::INFO 5739363 packets received on interface ens1f0, 0 dropped
1534891115.924612 Reporter::INFO 6341512 packets received on interface ens1f0, 0 dropped
1534891115.930012 Reporter::INFO 7133185 packets received on interface ens1f0, 0 dropped
1534891115.926888 Reporter::INFO 7315668 packets received on interface ens1f0, 0 dropped
1534891115.941661 Reporter::INFO 5996086 packets received on interface ens1f0, 0 dropped
1534891115.935347 Reporter::INFO 7124354 packets received on interface ens1f0, 0 dropped
1534891115.932771 Reporter::INFO 4812550 packets received on interface ens1f0, 0 dropped
1534891115.938545 Reporter::INFO 6444252 packets received on interface ens1f0, 0 dropped
1534891115.936348 Reporter::INFO 5005644 packets received on interface ens1f0, 0 dropped
1534891115.916074 Reporter::INFO 5041262 packets received on interface ens1f0, 0 dropped
1534891115.939653 Reporter::INFO 5810241 packets received on interface ens1f0, 0 dropped
1534891115.940698 Reporter::INFO 5284307 packets received on interface ens1f0, 0 dropped
1534891115.917379 Reporter::INFO 5701186 packets received on interface ens1f0, 0 dropped
1534891115.937405 Reporter::INFO 4785317 packets received on interface ens1f0, 0 dropped
1531828134.620284 Reporter::INFO 424506 packets received on interface eno1, 0 dropped
-----Original Message-----
From: Azoff, Justin S <jazoff at illinois.edu>
Sent: Wednesday, August 22, 2018 10:59 AM
To: Ron McClellan <Ron_McClellan at ao.uscourts.gov>
Cc: bro at bro.org
Subject: Re: [Bro] BRO Logger crashing due to large DNS log files
> On Aug 22, 2018, at 10:48 AM, Ron McClellan <Ron_McClellan at ao.uscourts.gov> wrote:
>
> Justin,
>
> Got good news and solid progress with your help. BRO is running on both boxes and hasn't crashed since 10pm last night. If I read the data about NUMA from my systems, I don't really need to split the load between 2 workers as you did, right?
If you can get another NIC so each box has 2, then you could divide the workers between each NIC and NUMA node. Otherwise it doesn't matter so much.
> I'm working on tuning some now and also trying to address the really high lag (500) that I'm still seeing. Currently seeing some loss on it, but will continue to tune and see what if I can get that under control. Let me know if you need help testing the doctor script.
>
> Ron
>
> 1534948572.682908 900.000005 worker-1-8 60904 532647 11.434214
> 1534948572.692674 900.000072 worker-1-13 67152 216975 30.949188
> 1534948572.688750 900.000028 worker-1-18 70383 235710 29.859997
> 1534948572.705484 900.000037 worker-1-24 57008 201189 28.335545
> 1534948572.682147 900.000099 worker-1-5 61878 194825 31.760811
> 1534948572.699536 900.000061 worker-1-16 76385 256671 29.759887
> 1534948572.682829 900.000080 worker-1-29 52464 188150 27.884135
> 1534948572.683536 900.000049 worker-1-4 110222 314119 35.08925
>
> [root at aosoc current]# broctl netstats
> worker-1-1: 1534949053.166850 recvd=813997 dropped=0 link=813997
> worker-1-2: 1534949053.366803 recvd=873351 dropped=0 link=873353
> worker-1-3: 1534949053.567778 recvd=1770808 dropped=0 link=1770810
> worker-1-4: 1534949053.767852 recvd=865443 dropped=0 link=865449
> worker-1-5: 1534949053.968873 recvd=349355 dropped=0 link=349361
> worker-1-6: 1534949054.168785 recvd=1152160 dropped=0 link=1152161
> worker-1-7: 1534949054.368825 recvd=1358553 dropped=0 link=1358553
> worker-1-8: 1534949054.569808 recvd=345267 dropped=0 link=345272
> worker-1-9: 1534949054.769982 recvd=856725 dropped=0 link=856732
> worker-1-10: 1534949054.969811 recvd=351148 dropped=0 link=351148
> worker-1-11: 1534949055.170855 recvd=883897 dropped=0 link=883897
> worker-1-12: 1534949055.370950 recvd=820117 dropped=0 link=820125
> worker-1-13: 1534949055.571899 recvd=1132465 dropped=0 link=1132473
> worker-1-14: 1534949055.771751 recvd=823249 dropped=0 link=823249
> worker-1-15: 1534949055.972921 recvd=754342 dropped=0 link=754343
> worker-1-16: 1534949056.173778 recvd=822102 dropped=0 link=822106
> worker-1-17: 1534949056.373806 recvd=570905 dropped=0 link=570911
> worker-1-18: 1534949056.573815 recvd=1033845 dropped=0 link=1033846
> worker-1-19: 1534949056.774737 recvd=648977 dropped=0 link=649001
> worker-1-20: 1534949056.974823 recvd=816836 dropped=0 link=816838
> worker-1-21: 1534949057.175858 recvd=423896 dropped=0 link=423901
> worker-1-22: 1534949057.375894 recvd=761794 dropped=0 link=761796
> worker-1-23: 1534949057.576737 recvd=415151 dropped=0 link=415153
> worker-1-24: 1534949057.776887 recvd=604342 dropped=0 link=604349
> worker-1-25: 1534949057.978046 recvd=911772 dropped=0 link=911785
> worker-1-26: 1534949058.177749 recvd=358386 dropped=0 link=358395
> worker-1-27: 1534949058.379062 recvd=1283463 dropped=0 link=1283465
> worker-1-28: 1534949058.578751 recvd=364801 dropped=0 link=364807
> worker-1-29: 1534949058.778735 recvd=930041 dropped=0 link=930042
> worker-1-30: 1534949058.979938 recvd=857963 dropped=0 link=857967
If you're seeing a high percentage of capture loss but netstats is showing 0 dropped packets that means one of two things:
* Something still isn't right with the load balancing. It could be that your NIC isn't doing symmetric hashing properly.
* There's an issue with the traffic upstream of bro.
A bunch of the checks that bro-doctor does can help diagnose this, but you'd need to re-enable the conn.log
—
Justin Azoff
More information about the Bro
mailing list