[Bro] BRO Logger crashing due to large DNS log files

Ron McClellan Ron_McClellan at ao.uscourts.gov
Tue Aug 21 15:10:58 PDT 2018


Justin,

	I finished most of your recommendations, just need to rebuild bro, but was going to let it run over night and see how it is running now.  I really appreciate all the help.

Thanks,

Ron



]# hwloc-ls -p
Machine (256GB total)
  NUMANode P#0 (128GB)
    Package P#0 + L3 (25MB)
      L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#0
        PU P#0
        PU P#36
      L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#1
        PU P#1
        PU P#37
      L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#2
        PU P#2
        PU P#38
      L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#3
        PU P#3
        PU P#39
      L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#4
        PU P#4
        PU P#40
      L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#8
        PU P#5
        PU P#41
      L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#9
        PU P#6
        PU P#42
      L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#10
        PU P#7
        PU P#43
      L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#11
        PU P#8
        PU P#44
      L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#16
        PU P#9
        PU P#45
      L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#17
        PU P#10
        PU P#46
      L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#18
        PU P#11
        PU P#47
      L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#19
        PU P#12
        PU P#48
      L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#20
        PU P#13
        PU P#49
      L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#24
        PU P#14
        PU P#50
      L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#25
        PU P#15
        PU P#51
      L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#26
        PU P#16
        PU P#52
      L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#27
        PU P#17
        PU P#53
    HostBridge P#0
      PCIBridge
        PCI 14e4:1657
          Net "eno1"
        PCI 14e4:1657
          Net "eno2"
        PCI 14e4:1657
          Net "eno3"
        PCI 14e4:1657
          Net "eno4"
      PCIBridge
        PCI 102b:0538
          GPU "card0"
          GPU "controlD64"
    HostBridge P#1
      PCIBridge
        PCI 8086:1572
          Net "ens1f0"
        PCI 8086:1572
          Net "ens1f1"
      PCIBridge
        PCI 8086:1572
          Net "ens3f0"
        PCI 8086:1572
          Net "ens3f1"
    HostBridge P#2
      PCIBridge
        PCI 8086:1572
          Net "ens2f0"
        PCI 8086:1572
          Net "ens2f1"
    HostBridge P#3
      PCIBridge
        PCI 9005:028f
          Block(Disk) "sda"
          Block(Disk) "sdc"
  NUMANode P#1 (128GB) + Package P#1 + L3 (25MB)
    L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#0
      PU P#18
      PU P#54
    L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#1
      PU P#19
      PU P#55
    L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#2
      PU P#20
      PU P#56
    L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#3
      PU P#21
      PU P#57
    L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#4
      PU P#22
      PU P#58
    L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#8
      PU P#23
      PU P#59
    L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#9
      PU P#24
      PU P#60
    L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#10
      PU P#25
      PU P#61
    L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#11
      PU P#26
      PU P#62
    L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#16
      PU P#27
      PU P#63
    L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#17
      PU P#28
      PU P#64
    L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#18
      PU P#29
      PU P#65
    L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#19
      PU P#30
      PU P#66
    L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#20
      PU P#31
      PU P#67
    L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#24
      PU P#32
      PU P#68
    L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#25
      PU P#33
      PU P#69
    L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#26
      PU P#34
      PU P#70
    L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#27
      PU P#35
      PU P#71

-----Original Message-----
From: Azoff, Justin S <jazoff at illinois.edu> 
Sent: Tuesday, August 21, 2018 5:53 PM
To: Ron McClellan <Ron_McClellan at ao.uscourts.gov>
Cc: bro at bro.org
Subject: Re: [Bro] BRO Logger crashing due to large DNS log files


> On Aug 21, 2018, at 4:43 PM, Azoff, Justin S <jazoff at illinois.edu> wrote:
> 
> You may also have 2 NUMA nodes in that case you would probably want to use 15 workers on each NUMA node, which would be something like
> 3-18 and 21-36... very tricky.  Ideally broctl could just support pin_cpus=auto as well :-)

I double checked this to verify I was saying the right thing, and looking closer at all our boxes that are NUMA it seems the cpus get allocated in round robin between them, so

0,2,4,6,8,10....34 go to numa node 0, then
1,3,5,7,9,11....35 go to numa node 1

But I think that can depend on kernel version and I swear this changed between centos6 and 7.

This matters more if you have 2 cards you can capture from though, since most likely one card is attached to each numa node.
If you only have one 10g interface you are capturing from it doesn't matter as much.

$ hwloc-ls -p
Machine (64GB total)
 NUMANode P#0 (32GB)
   Package P#0 + L3 (14MB)
     L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#0
       PU P#0
       PU P#20
     L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#4
       PU P#2
       PU P#22
     L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#1
       PU P#4
       PU P#24
     L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#3
       PU P#6
       PU P#26
     L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#2
       PU P#8
       PU P#28
     L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#12
       PU P#10
       PU P#30
     L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#8
       PU P#12
       PU P#32
     L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#11
       PU P#14
       PU P#34
     L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#9
       PU P#16
       PU P#36
     L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#10
       PU P#18
       PU P#38
   HostBridge P#2
     PCIBridge
       PCI 8086:1584
         Net "p2p1"
 NUMANode P#1 (32GB)
   Package P#1 + L3 (14MB)
     L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#0
       PU P#1
       PU P#21
     L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#4
       PU P#3
       PU P#23
     L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#1
       PU P#5
       PU P#25
     L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#3
       PU P#7
       PU P#27
     L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#2
       PU P#9
       PU P#29
     L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#12
       PU P#11
       PU P#31
     L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#8
       PU P#13
       PU P#33
     L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#11
       PU P#15
       PU P#35
     L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#9
       PU P#17
       PU P#37
     L2 (1024KB) + L1d (32KB) + L1i (32KB) + Core P#10
       PU P#19
       PU P#39
   HostBridge P#6
     PCIBridge
       PCI 8086:1584
         Net "p3p1"

This gets really confusing but it shows that p2p1 is connected to the first numa node and has real cpus 0,2,4,8... and hyperthread cores 20,22,24,26...

For boxes like that, the configuration would look something like

[...p2p1]
type=worker
host=..
interface=af_packet::p2p1
lb_method=custom
lb_procs=8
pin_cpus=4,6,8,10,12,14,16,18
af_packet_fanout_id=21
af_packet_fanout_mode=AF_Packet::FANOUT_HASH

[...p3p1]
type=worker
host=..
interface=af_packet::p3p1
lb_method=custom
lb_procs=8
pin_cpus=5,7,9,11,13,15,17,19
af_packet_fanout_id=22
af_packet_fanout_mode=AF_Packet::FANOUT_HASH



— 
Justin Azoff




More information about the Bro mailing list