[Bro] BRO Logger crashing due to large DNS log files
Ron McClellan
Ron_McClellan at ao.uscourts.gov
Wed Aug 22 18:00:36 PDT 2018
Justin,
I disabled checksum checking, but still get the same error in the doctor output. I'm not sure if the interfacesetup config is working, using defaults for now. I'm going to get with our network guy and review the tap configurations tomorrow and make sure there is no issues there.
Thanks Again,
Ron
## Global BroControl configuration file.
interfacesetup.enabled=1
#To change the default mtu that is configured
#interfacesetup.mtu=9000
#To change the default commands that are used to bring up the interface
#interfacesetup.up_command=/sbin/ifconfig {interface} up mtu {mtu}
#interfacesetup.flags_command=/sbin/ethtool -K {interface} gro off lro off rx off tx off gso off
## If true, don't verify checksums. Useful for running on altered trace
## files, and for saving a few cycles, but at the risk of analyzing invalid
## data. Note that the ``-C`` command-line option overrides the setting of this
## variable.
const ignore_checksums = T &redef;
#################################################################
# Checking if many recent connections have a SAD or had history #
#################################################################
error: 74.24%, 18466 out of 24872 connections are half duplex
################################################
# Checking for recent capture_loss.log entries #
################################################
Capture loss stats:
error: worker=worker-1-1 loss_count=103 noloss_count=0 min_loss=0.794921 max_loss=90.58044 overall_loss=27.2273066532
error: worker=worker-1-10 loss_count=102 noloss_count=0 min_loss=0.448354 max_loss=66.577529 overall_loss=23.5837489489
error: worker=worker-1-11 loss_count=102 noloss_count=0 min_loss=0.720309 max_loss=67.87389 overall_loss=26.0957227921
error: worker=worker-1-12 loss_count=102 noloss_count=0 min_loss=0.130618 max_loss=77.820137 overall_loss=24.5815538385
error: worker=worker-1-13 loss_count=102 noloss_count=0 min_loss=0.361211 max_loss=67.574163 overall_loss=24.8552395812
error: worker=worker-1-14 loss_count=102 noloss_count=0 min_loss=0.368082 max_loss=63.196486 overall_loss=27.4975008733
error: worker=worker-1-15 loss_count=102 noloss_count=0 min_loss=0.191227 max_loss=69.743213 overall_loss=26.8516883949
error: worker=worker-1-16 loss_count=102 noloss_count=0 min_loss=0.148612 max_loss=65.566384 overall_loss=25.6185453556
error: worker=worker-1-17 loss_count=102 noloss_count=0 min_loss=0.327409 max_loss=66.476362 overall_loss=26.3990676615
error: worker=worker-1-18 loss_count=102 noloss_count=0 min_loss=0.734925 max_loss=64.410006 overall_loss=24.4351738027
error: worker=worker-1-19 loss_count=102 noloss_count=0 min_loss=0.205711 max_loss=63.185604 overall_loss=25.5207097775
error: worker=worker-1-2 loss_count=102 noloss_count=0 min_loss=0.62053 max_loss=69.678688 overall_loss=27.4164344389
error: worker=worker-1-20 loss_count=102 noloss_count=0 min_loss=0.28481 max_loss=63.994922 overall_loss=22.7225619728
error: worker=worker-1-21 loss_count=102 noloss_count=0 min_loss=0.376553 max_loss=65.539711 overall_loss=24.2161148411
error: worker=worker-1-22 loss_count=103 noloss_count=0 min_loss=0.71613 max_loss=69.499069 overall_loss=29.2418154822
error: worker=worker-1-23 loss_count=102 noloss_count=0 min_loss=0.397382 max_loss=64.149083 overall_loss=28.2197289348
error: worker=worker-1-24 loss_count=102 noloss_count=0 min_loss=0.681759 max_loss=66.324143 overall_loss=21.3073056893
error: worker=worker-1-25 loss_count=102 noloss_count=0 min_loss=0.33738 max_loss=69.687833 overall_loss=26.2137549719
error: worker=worker-1-26 loss_count=102 noloss_count=0 min_loss=0.387409 max_loss=65.863473 overall_loss=26.7663244831
error: worker=worker-1-27 loss_count=102 noloss_count=0 min_loss=0.574677 max_loss=65.497259 overall_loss=26.6219106573
error: worker=worker-1-28 loss_count=102 noloss_count=0 min_loss=0.146595 max_loss=64.144564 overall_loss=27.6602933778
error: worker=worker-1-29 loss_count=102 noloss_count=0 min_loss=0.22543 max_loss=64.903986 overall_loss=27.0578623844
error: worker=worker-1-3 loss_count=102 noloss_count=0 min_loss=0.554694 max_loss=68.130508 overall_loss=25.1314879388
error: worker=worker-1-30 loss_count=102 noloss_count=0 min_loss=0.523326 max_loss=72.621044 overall_loss=23.007387178
error: worker=worker-1-4 loss_count=102 noloss_count=0 min_loss=0.604213 max_loss=67.538442 overall_loss=27.8744586007
error: worker=worker-1-5 loss_count=102 noloss_count=0 min_loss=0.088526 max_loss=71.771296 overall_loss=26.4599103542
error: worker=worker-1-6 loss_count=102 noloss_count=0 min_loss=0.196425 max_loss=71.430447 overall_loss=23.4866215325
error: worker=worker-1-7 loss_count=102 noloss_count=0 min_loss=0.72676 max_loss=66.640978 overall_loss=26.7013322956
error: worker=worker-1-8 loss_count=102 noloss_count=0 min_loss=0.222815 max_loss=81.943911 overall_loss=27.3586324115
error: worker=worker-1-9 loss_count=103 noloss_count=0 min_loss=0.861021 max_loss=72.243329 overall_loss=29.50492059
################################################################
# Checking what percentage of recent tcp connections show loss #
################################################################
0.04%, 10 out of 24872 connections have capture loss
###################################################################
# Checking if connections are unevenly distributed across workers #
###################################################################
error: No node names in conn log. Install add-node-names package to add a corresponding field.
###############################################################################################################################
# Checking if anything is in the deprecated local-logger.bro, local-manager.bro, local-proxy.bro, or local-worker.bro scripts #
###############################################################################################################################
Nothing found
######################################################################
# Checking if any recent connections have been logged multiple times #
######################################################################
ok, only 0.00%, 0 out of 683 connections appear to be duplicate
############################################################################
# Checking what percentage of recent tcp connections are remote to remote. #
############################################################################
0.01%, 11 out of 100000 connections are remote to remote
###############################################################################
# Checking if bro is linked against a custom malloc like tcmalloc or jemalloc #
###############################################################################
error: configured to use a custom malloc=False
##################################
# Checking pf_ring configuration #
##################################
configured to use pf_ring=False pcap=False plugin=False
############################################
# Checking for recent reporter.log entries #
############################################
error: Found 22 reporter log files in the past 7 days
Recent reporter.log messages:
Reporter::INFO processing suspended
Reporter::INFO processing suspended
221 duplicate messages suppressed
Reporter::INFO processing continued
Reporter::INFO processing continued
268 duplicate messages suppressed
/usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro, line 54 1534944982.705403 Reporter::WARNING Your interface is likely receiving invalid TCP checksums, most likely from NIC checksum offloading. By default, packets with invalid checksums are discarded by Bro unless using the -C command-line option or toggling the 'ignore_checksums' variable. Alternatively, disable checksum offloading by the network adapter to ensure Bro analyzes the actual checksums that are transmitted.
/usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro, line 54 1534971018.604887 Reporter::WARNING Your interface is likely receiving invalid TCP checksums, most likely from NIC checksum offloading. By default, packets with invalid checksums are discarded by Bro unless using the -C command-line option or toggling the 'ignore_checksums' variable. Alternatively, disable checksum offloading by the network adapter to ensure Bro analyzes the actual checksums that are transmitted.
21 duplicate messages suppressed
/usr/local/bro/share/bro/policy/protocols/ssh/geo-data.bro, line 30 1534945001.027149 Reporter::ERROR Bro was not configured for GeoIP support (lookup_location(SSH::lookup_ip))
/usr/local/bro/share/bro/policy/protocols/ssh/geo-data.bro, line 30 1534945023.085730 Reporter::ERROR Bro was not configured for GeoIP support (lookup_location(SSH::lookup_ip))
95 duplicate messages suppressed
1534944917.319061 Reporter::INFO received termination signal
1534944917.325143 Reporter::INFO received termination signal
196 duplicate messages suppressed
1534944917.319061 Reporter::INFO 59439914 packets received on interface ens1f0, 0 dropped
1534944917.325143 Reporter::INFO 45022136 packets received on interface ens1f0, 0 dropped
1534944917.321217 Reporter::INFO 51627314 packets received on interface ens1f0, 0 dropped
1534944917.322883 Reporter::INFO 53891270 packets received on interface ens1f0, 0 dropped
1534944917.324085 Reporter::INFO 88417370 packets received on interface ens1f0, 0 dropped
1534944917.319957 Reporter::INFO 54791897 packets received on interface ens1f0, 0 dropped
1534944917.327305 Reporter::INFO 43678446 packets received on interface ens1f0, 0 dropped
1534944917.330087 Reporter::INFO 59703116 packets received on interface ens1f0, 0 dropped
1534944917.328208 Reporter::INFO 45211345 packets received on interface ens1f0, 0 dropped
1534944917.331134 Reporter::INFO 65262767 packets received on interface ens1f0, 0 dropped
1534944917.334825 Reporter::INFO 55188270 packets received on interface ens1f0, 0 dropped
1534944917.329257 Reporter::INFO 52034434 packets received on interface ens1f0, 0 dropped
1534944917.332107 Reporter::INFO 49495024 packets received on interface ens1f0, 0 dropped
1534944917.333015 Reporter::INFO 49204592 packets received on interface ens1f0, 0 dropped
1534944917.336672 Reporter::INFO 50396962 packets received on interface ens1f0, 0 dropped
1534944917.333880 Reporter::INFO 54559564 packets received on interface ens1f0, 0 dropped
1534944917.338831 Reporter::INFO 55635174 packets received on interface ens1f0, 0 dropped
1534944917.341397 Reporter::INFO 58206831 packets received on interface ens1f0, 0 dropped
1534944917.344880 Reporter::INFO 56679728 packets received on interface ens1f0, 0 dropped
1534944917.343923 Reporter::INFO 50649583 packets received on interface ens1f0, 0 dropped
1534944917.345953 Reporter::INFO 69599231 packets received on interface ens1f0, 0 dropped
1534944917.340038 Reporter::INFO 52187914 packets received on interface ens1f0, 0 dropped
1534944917.342584 Reporter::INFO 54560034 packets received on interface ens1f0, 0 dropped
1534944917.346950 Reporter::INFO 43540603 packets received on interface ens1f0, 0 dropped
1534944917.351083 Reporter::INFO 94170430 packets received on interface ens1f0, 0 dropped
1534944917.326276 Reporter::INFO 61679606 packets received on interface ens1f0, 0 dropped
1534944917.348123 Reporter::INFO 52021287 packets received on interface ens1f0, 0 dropped
1534944917.350083 Reporter::INFO 59524072 packets received on interface ens1f0, 0 dropped
1534944917.349146 Reporter::INFO 53404009 packets received on interface ens1f0, 0 dropped
1534944917.337775 Reporter::INFO 53757378 packets received on interface ens1f0, 0 dropped
1534981838.348667 Reporter::INFO 800278 packets received on interface ens1f0, 0 dropped
1534981838.349759 Reporter::INFO 1300570 packets received on interface ens1f0, 0 dropped
1534981838.352160 Reporter::INFO 1038523 packets received on interface ens1f0, 0 dropped
1534981838.353313 Reporter::INFO 602996 packets received on interface ens1f0, 0 dropped
1534981838.357869 Reporter::INFO 595341 packets received on interface ens1f0, 0 dropped
1534981838.358662 Reporter::INFO 630136 packets received on interface ens1f0, 0 dropped
1534981838.350883 Reporter::INFO 558995 packets received on interface ens1f0, 0 dropped
1534981838.360612 Reporter::INFO 647854 packets received on interface ens1f0, 0 dropped
1534981838.355404 Reporter::INFO 742801 packets received on interface ens1f0, 0 dropped
1534981838.354432 Reporter::INFO 1158902 packets received on interface ens1f0, 0 dropped
1534981838.356568 Reporter::INFO 830080 packets received on interface ens1f0, 0 dropped
1534981838.359734 Reporter::INFO 758136 packets received on interface ens1f0, 0 dropped
1534981838.367822 Reporter::INFO 550137 packets received on interface ens1f0, 0 dropped
1534981838.361739 Reporter::INFO 665719 packets received on interface ens1f0, 0 dropped
1534981838.362719 Reporter::INFO 488230 packets received on interface ens1f0, 0 dropped
1534981838.365881 Reporter::INFO 826799 packets received on interface ens1f0, 0 dropped
1534981838.363606 Reporter::INFO 524107 packets received on interface ens1f0, 0 dropped
1534981838.366718 Reporter::INFO 1593921 packets received on interface ens1f0, 0 dropped
1534981838.368750 Reporter::INFO 631846 packets received on interface ens1f0, 0 dropped
1534981838.364730 Reporter::INFO 555121 packets received on interface ens1f0, 0 dropped
1534981838.369824 Reporter::INFO 610239 packets received on interface ens1f0, 0 dropped
1534981838.371295 Reporter::INFO 575919 packets received on interface ens1f0, 0 dropped
1534981838.373749 Reporter::INFO 1054019 packets received on interface ens1f0, 0 dropped
1534981838.372598 Reporter::INFO 557356 packets received on interface ens1f0, 0 dropped
1534981838.376138 Reporter::INFO 1123785 packets received on interface ens1f0, 0 dropped
1534981838.379690 Reporter::INFO 668758 packets received on interface ens1f0, 0 dropped
1534981838.377535 Reporter::INFO 1421582 packets received on interface ens1f0, 0 dropped
1534981838.378215 Reporter::INFO 669462 packets received on interface ens1f0, 0 dropped
1534981838.375390 Reporter::INFO 634210 packets received on interface ens1f0, 0 dropped
1534981838.380430 Reporter::INFO 537366 packets received on interface ens1f0, 0 dropped
1534981216.903010 Reporter::INFO 6762256 packets received on interface ens1f0, 0 dropped
1534981216.905617 Reporter::INFO 5045464 packets received on interface ens1f0, 0 dropped
1534981216.904381 Reporter::INFO 4320028 packets received on interface ens1f0, 0 dropped
1534981216.909996 Reporter::INFO 3877533 packets received on interface ens1f0, 0 dropped
1534981216.909067 Reporter::INFO 6006595 packets received on interface ens1f0, 0 dropped
1534981216.913121 Reporter::INFO 6123109 packets received on interface ens1f0, 0 dropped
1534981216.911284 Reporter::INFO 3797206 packets received on interface ens1f0, 0 dropped
1534981216.912225 Reporter::INFO 3539501 packets received on interface ens1f0, 0 dropped
1534981216.913347 Reporter::INFO 3215296 packets received on interface ens1f0, 0 dropped
1534981216.907955 Reporter::INFO 4181023 packets received on interface ens1f0, 0 dropped
1534981216.915150 Reporter::INFO 4228532 packets received on interface ens1f0, 0 dropped
1534981216.917084 Reporter::INFO 3895338 packets received on interface ens1f0, 0 dropped
1534981216.919152 Reporter::INFO 4093951 packets received on interface ens1f0, 0 dropped
1534981216.916182 Reporter::INFO 4888118 packets received on interface ens1f0, 0 dropped
1534981216.930824 Reporter::INFO 4969258 packets received on interface ens1f0, 0 dropped
1534981216.923402 Reporter::INFO 3541359 packets received on interface ens1f0, 0 dropped
1534981216.921326 Reporter::INFO 5378841 packets received on interface ens1f0, 0 dropped
1534981216.918085 Reporter::INFO 3692685 packets received on interface ens1f0, 0 dropped
1534981216.927073 Reporter::INFO 4201218 packets received on interface ens1f0, 0 dropped
1534981216.929713 Reporter::INFO 2972462 packets received on interface ens1f0, 0 dropped
1534981216.920093 Reporter::INFO 3657035 packets received on interface ens1f0, 0 dropped
1534981216.934715 Reporter::INFO 3807340 packets received on interface ens1f0, 0 dropped
1534981216.924635 Reporter::INFO 4475718 packets received on interface ens1f0, 0 dropped
1534981216.936953 Reporter::INFO 4226114 packets received on interface ens1f0, 0 dropped
1534981216.928142 Reporter::INFO 5128429 packets received on interface ens1f0, 0 dropped
1534981216.925807 Reporter::INFO 5599345 packets received on interface ens1f0, 0 dropped
1534981216.906854 Reporter::INFO 3459585 packets received on interface ens1f0, 0 dropped
1534981216.935880 Reporter::INFO 4798353 packets received on interface ens1f0, 0 dropped
1534981216.933520 Reporter::INFO 3658076 packets received on interface ens1f0, 0 dropped
1534981216.931230 Reporter::INFO 3996410 packets received on interface ens1f0, 0 dropped
1534970997.269085 Reporter::INFO 75579541 packets received on interface ens1f0, 0 dropped
1534970997.272310 Reporter::INFO 77450480 packets received on interface ens1f0, 0 dropped
1534970997.275471 Reporter::INFO 73248332 packets received on interface ens1f0, 0 dropped
1534970997.270934 Reporter::INFO 91638422 packets received on interface ens1f0, 0 dropped
1534970997.273268 Reporter::INFO 98209619 packets received on interface ens1f0, 0 dropped
1534970997.277737 Reporter::INFO 102107215 packets received on interface ens1f0, 0 dropped
1534970997.274526 Reporter::INFO 80458937 packets received on interface ens1f0, 0 dropped
1534970997.278958 Reporter::INFO 76230406 packets received on interface ens1f0, 0 dropped
1534970997.276604 Reporter::INFO 80722958 packets received on interface ens1f0, 0 dropped
1534970997.279882 Reporter::INFO 100099311 packets received on interface ens1f0, 0 dropped
1534970997.283074 Reporter::INFO 69951029 packets received on interface ens1f0, 0 dropped
1534970997.281120 Reporter::INFO 75612600 packets received on interface ens1f0, 0 dropped
1534970997.282298 Reporter::INFO 77578375 packets received on interface ens1f0, 0 dropped
1534970997.284196 Reporter::INFO 85732570 packets received on interface ens1f0, 0 dropped
1534970997.289385 Reporter::INFO 76071057 packets received on interface ens1f0, 0 dropped
1534970997.287212 Reporter::INFO 82814054 packets received on interface ens1f0, 0 dropped
1534970997.285008 Reporter::INFO 86418433 packets received on interface ens1f0, 0 dropped
1534970997.286085 Reporter::INFO 81872671 packets received on interface ens1f0, 0 dropped
1534970997.288381 Reporter::INFO 80730232 packets received on interface ens1f0, 0 dropped
1534970997.290577 Reporter::INFO 82468402 packets received on interface ens1f0, 0 dropped
1534970997.294493 Reporter::INFO 83079761 packets received on interface ens1f0, 0 dropped
1534970997.291726 Reporter::INFO 75226715 packets received on interface ens1f0, 0 dropped
1534970997.301214 Reporter::INFO 73921183 packets received on interface ens1f0, 0 dropped
1534970997.295776 Reporter::INFO 76411550 packets received on interface ens1f0, 0 dropped
1534970997.293085 Reporter::INFO 78806075 packets received on interface ens1f0, 0 dropped
1534970997.299137 Reporter::INFO 92333460 packets received on interface ens1f0, 0 dropped
1534970997.296879 Reporter::INFO 76489847 packets received on interface ens1f0, 0 dropped
1534970997.297955 Reporter::INFO 77040177 packets received on interface ens1f0, 0 dropped
1534970997.300228 Reporter::INFO 73375388 packets received on interface ens1f0, 0 dropped
1534970997.270025 Reporter::INFO 85569417 packets received on interface ens1f0, 0 dropped
1534977985.775660 Reporter::INFO 23439044 packets received on interface ens1f0, 0 dropped
1534977985.774108 Reporter::INFO 17765795 packets received on interface ens1f0, 0 dropped
1534977985.777926 Reporter::INFO 16760996 packets received on interface ens1f0, 0 dropped
1534977985.776700 Reporter::INFO 17259361 packets received on interface ens1f0, 0 dropped
1534977985.780145 Reporter::INFO 18713880 packets received on interface ens1f0, 0 dropped
1534977985.779011 Reporter::INFO 16901868 packets received on interface ens1f0, 0 dropped
1534977985.781391 Reporter::INFO 18714893 packets received on interface ens1f0, 0 dropped
1534977985.783404 Reporter::INFO 16282338 packets received on interface ens1f0, 0 dropped
1534977985.782445 Reporter::INFO 16293375 packets received on interface ens1f0, 0 dropped
1534977985.784534 Reporter::INFO 29243070 packets received on interface ens1f0, 0 dropped
1534977985.786378 Reporter::INFO 23601045 packets received on interface ens1f0, 0 dropped
1534977985.790525 Reporter::INFO 20118319 packets received on interface ens1f0, 0 dropped
1534977985.785406 Reporter::INFO 16852112 packets received on interface ens1f0, 0 dropped
1534977985.791658 Reporter::INFO 16997069 packets received on interface ens1f0, 0 dropped
1534977985.789483 Reporter::INFO 16957150 packets received on interface ens1f0, 0 dropped
1534977985.787405 Reporter::INFO 15663429 packets received on interface ens1f0, 0 dropped
1534977985.793724 Reporter::INFO 15732747 packets received on interface ens1f0, 0 dropped
1534977985.792848 Reporter::INFO 20594428 packets received on interface ens1f0, 0 dropped
1534977985.795803 Reporter::INFO 17450153 packets received on interface ens1f0, 0 dropped
1534977985.794824 Reporter::INFO 19299170 packets received on interface ens1f0, 0 dropped
1534977985.799558 Reporter::INFO 21344899 packets received on interface ens1f0, 0 dropped
1534977985.797155 Reporter::INFO 15890636 packets received on interface ens1f0, 0 dropped
1534977985.803070 Reporter::INFO 16998576 packets received on interface ens1f0, 0 dropped
1534977985.798511 Reporter::INFO 15619707 packets received on interface ens1f0, 0 dropped
1534977985.799711 Reporter::INFO 17108695 packets received on interface ens1f0, 0 dropped
1534977985.805260 Reporter::INFO 16192817 packets received on interface ens1f0, 0 dropped
1534977985.804186 Reporter::INFO 18282565 packets received on interface ens1f0, 0 dropped
1534977985.802072 Reporter::INFO 16240542 packets received on interface ens1f0, 0 dropped
1534977985.806260 Reporter::INFO 22287402 packets received on interface ens1f0, 0 dropped
1534977985.788479 Reporter::INFO 15441331 packets received on interface ens1f0, 0 dropped
1534978504.904141 Reporter::INFO 1151439 packets received on interface ens1f0, 0 dropped
1534978504.906757 Reporter::INFO 672616 packets received on interface ens1f0, 0 dropped
1534978504.910902 Reporter::INFO 798199 packets received on interface ens1f0, 0 dropped
1534978504.907730 Reporter::INFO 820256 packets received on interface ens1f0, 0 dropped
1534978504.905443 Reporter::INFO 1001287 packets received on interface ens1f0, 0 dropped
1534978504.908955 Reporter::INFO 1377270 packets received on interface ens1f0, 0 dropped
1534978504.911752 Reporter::INFO 1220954 packets received on interface ens1f0, 0 dropped
1534978504.912143 Reporter::INFO 1084331 packets received on interface ens1f0, 0 dropped
1534978504.914831 Reporter::INFO 1029296 packets received on interface ens1f0, 0 dropped
1534978504.916659 Reporter::INFO 912276 packets received on interface ens1f0, 0 dropped
1534978504.915751 Reporter::INFO 772154 packets received on interface ens1f0, 0 dropped
1534978504.918896 Reporter::INFO 696069 packets received on interface ens1f0, 0 dropped
1534978504.909908 Reporter::INFO 1059539 packets received on interface ens1f0, 0 dropped
1534978504.917650 Reporter::INFO 733904 packets received on interface ens1f0, 0 dropped
1534978504.913752 Reporter::INFO 1163756 packets received on interface ens1f0, 0 dropped
1534978504.921136 Reporter::INFO 788919 packets received on interface ens1f0, 0 dropped
1534978504.919927 Reporter::INFO 710985 packets received on interface ens1f0, 0 dropped
1534978504.923178 Reporter::INFO 819349 packets received on interface ens1f0, 0 dropped
1534978504.925485 Reporter::INFO 1297935 packets received on interface ens1f0, 0 dropped
1534978504.924478 Reporter::INFO 918820 packets received on interface ens1f0, 0 dropped
1534978504.922142 Reporter::INFO 735662 packets received on interface ens1f0, 0 dropped
1534978504.932603 Reporter::INFO 1145281 packets received on interface ens1f0, 0 dropped
1534978504.928078 Reporter::INFO 714078 packets received on interface ens1f0, 0 dropped
1534978504.902959 Reporter::INFO 696518 packets received on interface ens1f0, 0 dropped
1534978504.929356 Reporter::INFO 1289706 packets received on interface ens1f0, 0 dropped
1534978504.931471 Reporter::INFO 716412 packets received on interface ens1f0, 0 dropped
1534978504.930397 Reporter::INFO 803952 packets received on interface ens1f0, 0 dropped
1534978504.934880 Reporter::INFO 1615952 packets received on interface ens1f0, 0 dropped
1534978504.927006 Reporter::INFO 663539 packets received on interface ens1f0, 0 dropped
1534978504.933828 Reporter::INFO 857559 packets received on interface ens1f0, 0 dropped
1534891115.907765 Reporter::INFO 5219842 packets received on interface ens1f0, 0 dropped
1534891115.910010 Reporter::INFO 8455238 packets received on interface ens1f0, 0 dropped
1534891115.912626 Reporter::INFO 5228397 packets received on interface ens1f0, 0 dropped
1534891115.908858 Reporter::INFO 5747798 packets received on interface ens1f0, 0 dropped
1534891115.911476 Reporter::INFO 5697178 packets received on interface ens1f0, 0 dropped
1534891115.915011 Reporter::INFO 4743597 packets received on interface ens1f0, 0 dropped
1534891115.918402 Reporter::INFO 6230492 packets received on interface ens1f0, 0 dropped
1534891115.913823 Reporter::INFO 7316772 packets received on interface ens1f0, 0 dropped
1534891115.920544 Reporter::INFO 8210898 packets received on interface ens1f0, 0 dropped
-----Original Message-----
From: Azoff, Justin S <jazoff at illinois.edu>
Sent: Wednesday, August 22, 2018 1:01 PM
To: Ron McClellan <Ron_McClellan at ao.uscourts.gov>
Cc: bro at bro.org
Subject: Re: [Bro] BRO Logger crashing due to large DNS log files
> On Aug 22, 2018, at 11:40 AM, Ron McClellan <Ron_McClellan at ao.uscourts.gov> wrote:
>
> Sorry, forgot to send that, I did re-enable the conn.log.
>
> Ron
>
A few things look like they are still not right.
> /usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro, line 54 1534945591.937813 Reporter::WARNING Your interface is likely receiving invalid TCP checksums, most likely from NIC checksum offloading. By default, packets with invalid checksums are discarded by Bro unless using the -C command-line option or toggling the 'ignore_checksums' variable. Alternatively, disable checksum offloading by the network adapter to ensure Bro analyzes the actual checksums that are transmitted.
> /usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro, line 54 1534891136.995923 Reporter::WARNING Your interface is likely receiving invalid TCP checksums, most likely from NIC checksum offloading. By default, packets with invalid checksums are discarded by Bro unless using the -C command-line option or toggling the 'ignore_checksums' variable. Alternatively, disable checksum offloading by the network adapter to ensure Bro analyzes the actual checksums that are transmitted.
> 5 duplicate messages suppressed
Ensuring all the offloading is setup correctly on your interfaces may fix a few of these. I have another plugin for broctl that can do this automatically for you:
https://packages.bro.org/packages/view/7520ca9d-4fb7-11e8-88be-0a645a3f3086
you just need to install it and then add
interfacesetup.enabled=1
to your broctl.cfg
The next time bro restarts it will ensure a bunch of settings are set properly using ethtool.
> #################################################################
> # Checking if many recent connections have a SAD or had history #
> #################################################################
> error: 24.30%, 52 out of 214 connections are half duplex
>
>
> #################################################################
> # Checking if many recent connections have a SAD or had history #
> #################################################################
> error: 75.53%, 14289 out of 18918 connections are half duplex
These 2 show that things are not working well at all. The rest of the checks don't mean much until this is fixed, so this is the thing to focus on.
This could be caused by an internal load balancing problem, or by an upstream issue. It's pretty easy to figure out which, I just haven't worked out the best way to have bro-doctor automate it.
What you want to do is run this script from a host that bro will see the request and response:
for x in $(seq -w 1 19); do
echo -e 'GET / HTTP/1.1\r\nHost: www.bro.org\r\n\r\n' |
socat - tcp-connect:www.bro.org:80,sp=300$x,reuseaddr;
sleep 1;
done
Wait a few minutes (if packets are being dropped, it may take 5 minutes for the log entries to show up only after the connections timeout) and then run
cat conn.log |bro-cut -d ts id.orig_p id.resp_h id.resp_p history orig_pkts resp_pkts|fgrep 192.150.187.43|fgrep 300
You should see something like this:
2018-08-22T11:53:46-0500 30001 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:53:47-0500 30002 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:53:48-0500 30003 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:53:49-0500 30004 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:53:50-0500 30005 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:53:52-0500 30006 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:53:53-0500 30007 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:53:54-0500 30008 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:53:55-0500 30009 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:53:56-0500 30010 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:53:57-0500 30011 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:53:59-0500 30012 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:54:00-0500 30013 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:54:01-0500 30014 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:54:02-0500 30015 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:54:03-0500 30016 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:54:04-0500 30017 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:54:06-0500 30018 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:54:07-0500 30019 192.150.187.43 80 ShADFadf 6 4
duplicate, missing, or split entries indicate different problems.
>
> ######################################################################
> ###### # Checking what percentage of recent tcp connections are remote
> to remote. #
> ######################################################################
> ######
> error: 52.85%, 52853 out of 100000 connections are remote to remote
In your networks.cfg make sure you have listed all of your 'local' address space.
—
Justin Azoff
More information about the Bro
mailing list