[Bro] Bro 2.3.2-419 segfaults when using PF_RING 6.0.3 libpcap 1.6.2 and pfdnacluster_master on RHEL 6.6

Thu Feb 26 16:37:30 PST 2015

Confirmed, pf_ring from SVN (with libpcap 1.6.x line), Bro 2.3.2 and
no segfaults from over 10 hours now.

Everything works in the AWS VM (so no DNA/ZC, but multiple workers). I
had issues with release version of pf_ring (6.0.2) but 6.0.3 build
9021 is OK.

On Thu, Feb 26, 2015 at 11:13 PM, Gary Faulkner <gfaulkner.nsm at gmail.com> wrote:
> All,
>
> A few other folks reported similar segfault issues to the PF_RING team
> both with standard PF_RING and DNA/ZC. After some troubleshooting and
> debugging they were able to to issue a patch (in SVN build 9021) that at
> least in initial testing seems to have resolved the segfault issue. Bro
> appears to now work segfault free using PF_RING (6.0.3 build 9021) both
> without DNA/ZC and with DNA using RSS. I'm still seeing a separate issue
> I'm following up with them on concerning not being able to map more than
> 10 app instances when using libzero's pfdnacluster_master script for
> load-balancing on host.
>
> Regards,
> Gary
>
> On 2/24/2015 2:59 PM, Gary Faulkner wrote:
>> A couple folks have suggested I run this with gdb and get a backtrace
>> to post here. Here is a quick gdb session with a backtrace of when I
>> run bro -i dnacluster:21 at 0:
>>
>> # gdb /nsm/bro/bin/bro
>> GNU gdb (GDB) SLES Expanded Support platform (7.2-75.el6)
>> Copyright (C) 2010 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.
>> html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.  Type "show
>> copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-redhat-linux".
>> For bug reporting instructions, please see:
>> <http://www.gnu.org/software/gdb/bugs/>...
>> Reading symbols from /nsm/bro/bin/bro...done.
>> (gdb) run -i dnacluster:21 at 0
>> Starting program: /nsm/bro/bin/bro -i dnacluster:21 at 0
>> [Thread debugging using libthread_db enabled]
>> listening on dnacluster:21 at 0, capture length 8192 bytes
>>
>> [New Thread 0x7fff20fd0700 (LWP 36513)]
>> [New Thread 0x7fff1bfff700 (LWP 36514)]
>> [New Thread 0x7fff1b5fe700 (LWP 36515)]
>> [New Thread 0x7fff1abfd700 (LWP 36516)]
>> [New Thread 0x7fff1a1fc700 (LWP 36517)]
>> [New Thread 0x7fff197fb700 (LWP 36518)]
>> [New Thread 0x7fff18dfa700 (LWP 36519)]
>> [New Thread 0x7fff03fff700 (LWP 36520)]
>> [New Thread 0x7fff035fe700 (LWP 36521)]
>> [New Thread 0x7fff02bfd700 (LWP 36522)]
>> [New Thread 0x7fff021fc700 (LWP 36523)]
>> [New Thread 0x7fff017fb700 (LWP 36524)]
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> 0x00007ffff7959506 in pcap_read_packet (handle=0x2631640,
>> callback=0x7ffff795d720 <pcap_oneshot>, userdata=0x7fffffffda20
>> "p\025c\002") at ./pcap-linux.c:1807
>> 1807    ./pcap-linux.c: No such file or directory.
>>         in ./pcap-linux.c
>> Missing separate debuginfos, use: debuginfo-install
>> GeoIP-1.5.1-5.el6.x86_64 glibc-2.12-1.149.el6_6.5.x86_64
>> keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-33.el6.x86_64
>> libcom_err-1.41.12-21.el6.x86_64 libgcc-4.4.7-11.el6.x86_64
>> libselinux-2.0.94-5.8.el6.x86_64 libstdc++-4.4.7-11.el6.x86_64
>> numactl-2.0.9-2.el6.x86_64 openssl-1.0.1e-30.el6_6.5.x86_64
>> zlib-1.2.3-29.el6.x86_64
>> (gdb) bt
>> #0  0x00007ffff7959506 in pcap_read_packet (handle=0x2631640,
>> callback=0x7ffff795d720 <pcap_oneshot>, userdata=0x7fffffffda20
>> "p\025c\002") at ./pcap-linux.c:1807
>> #1  0x00007ffff795d79b in pcap_next (p=<value optimized out>, h=<value
>> optimized out>) at ./pcap.c:218
>> #2  0x0000000000a4a490 in iosource::pcap::PcapSource::ExtractNextPacket
>> (this=0x2631430, pkt=0x2631468) at/nsm/bro/git/bro2.3-419/bro/
>> src/iosource/pcap/Source.cc:151
>> #3  0x0000000000a7580c in iosource::PktSrc::ExtractNextPacketInternal
>> (this=0x2631430) at
>> /nsm/bro/git/bro2.3-419/bro/src/iosource/PktSrc.cc:432
>> #4  0x0000000000a7511b in iosource::PktSrc::NextTimestamp
>> (this=0x2631430, local_network_time=0x7fffffffdcb8) at
>> /nsm/bro/git/bro2.3-419/bro/src/iosource/PktSrc.cc:241
>> #5  0x0000000000a71193 in iosource::Manager::FindSoonest (this=0xf29bc0,
>> ts=0x7fffffffddc8) at/nsm/bro/git/bro2.3-419/bro/
>> src/iosource/Manager.cc:82
>> #6  0x00000000007895d1 in net_run () at/nsm/bro/git/bro2.3-419/bro/
>> src/Net.cc:301
>> #7  0x00000000006d8ed7 in main (argc=3, argv=0x7fffffffe498) at
>> /nsm/bro/git/bro2.3-419/bro/src/main.cc:1200
>>
>> On 2/24/2015 1:20 PM, John Donnelly wrote:
>>>   Can you use gdb to get a backstrace ?
>>>
>>> ...
>>>
>>> ---------- Forwarded message ----------
>>> From: Gary Faulkner <gfaulkner.nsm at gmail.com>
>>> Date: Tue, Feb 24, 2015 at 12:23 PM
>>> Subject: [Bro] Bro 2.3.2-419 segfaults when using PF_RING 6.0.3 libpcap
>>> 1.6.2 and pfdnacluster_master on RHEL 6.6
>>> To: "bro at bro.org List" <bro at bro.org>
>>>
>>>
>>>   Hello,
>>>
>>> I’m having trouble getting Bro to run with PF_RING after updating
>>> from RHEL
>>> 6.5 to RHEL 6.6. The PF_RING aware drivers (DNA/ZC etc) in the “stable”
>>> 6.0.2 branch of PF_RING don’t appear to compile correctly on RHEL 6.6,
>>> which necessitated a move to the latest 6.0.3 development branch
>>> (rev.9009). This version compiles fine and I have it working with both
>>> Suricata and nprobe, but can’t get it working with Bro. Bro doesn’t
>>> seem to
>>> be able to open the dnacluster:21 at 0 etc interfaces with the new version.
>>> Specifically bro segfaults when calling the PF_RING version of
>>> libpcap.so.1.6.2, which is a new version of libpcap in 6.0.3. Previously
>>> libpcap was 1.1.1. I have also tried to compile PF_RING 6.0.2 stable on
>>> RHEL 6.6 with the newer drivers, but the version of pfdnacluster_master
>>> that ships with PF_RING 6.0.2 stable (that uses the older libpcap) will
>>> silently crash on RHEL 6.6. I’ve attached the output of a broctl diag to
>>> this email. Typically when I've seen an error where bro can’t listen on
>>> dnacluster in the past it has been due to the interface already being in
>>> use, bro not being able to find pfring, or not compiling against the
>>> correct libpcap. I’ve verified this isn’t the case to the best of my
>>> ability (no other libpcap on the system, fresh dna driver load and
>>> instance
>>> of pfdnaclster_master, pfring in $PATH etc). I’ve also verified that
>>> I can
>>> see packets on the dnacluster interfaces by testing with pfcount. It
>>> looks
>>> like perhaps bro doesn’t like the new version of libpcap. I have tried
>>> compiling and running bro with debugging enabled, but bro seems to
>>> crash on
>>> the workers without generating anything in the various debug.log
>>> files. Any
>>> thoughts?
>>>
>>> Here are example error messages from /var/log/messages:
>>>
>>> kernel: bro[1653]: segfault at 1371670 ip 00007f5a9e7f0660 sp
>>> 00007fff8714b300 error 4 in libpcap.so.1.6.2[7f5a9e7d9000+90000]
>>> kernel: bro[1643]: segfault at 1371670 ip 00007ff16d19b660 sp
>>> 00007fff81eea9a0 error 4 in libpcap.so.1.6.2[7ff16d184000+90000]
>>> kernel: bro[1656]: segfault at 1371670 ip 00007fcf3c6cf660 sp
>>> 00007fff3e1789b0 error 4 in libpcap.so.1.6.2[7fcf3c6b8000+90000]
>>> kernel: bro[1644]: segfault at 1 ip 00007f5932268506 sp 00007fffcd3ea0b0
>>> error 4 in libpcap.so.1.6.2[7f5932251000+90000]
>>> kernel: bro[1642]: segfault at 1 ip 00007ff3d1c83506 sp 00007fff468f4930
>>> error 4 in libpcap.so.1.6.2[7ff3d1c6c000+90000]
>>> kernel: bro[1658]: segfault at 1371670 ip 00007f53584f2660 sp
>>> 00007ffff89515f0 error 4 in libpcap.so.1.6.2[7f53584db000+90000]
>>> kernel: bro[1652]: segfault at 1371670 ip 00007f158fbc7660 sp
>>> 00007fff14aa7e20 error 4 in libpcap.so.1.6.2[7f158fbb0000+90000]
>>> kernel: bro[1660]: segfault at 1371670 ip 00007f2fee8e7660 sp
>>> 00007ffff9dacaf0 error 4 in libpcap.so.1.6.2[7f2fee8d0000+90000]
>>> kernel: bro[1641]: segfault at 1 ip 00007f32fbc48506 sp 00007fff7d9b2a00
>>> error 4 in libpcap.so.1.6.2[7f32fbc31000+90000]
>>> kernel: bro[1662]: segfault at b836210 ip 00007f5c9d669660 sp
>>> 00007fff71636fb0 error 4 in libpcap.so.1.6.2[7f5c9d652000+90000]
>>> kernel: bro[4220]: segfault at 1371670 ip 00007f6d35299660 sp
>>> 00007fff4d896940 error 4 in libpcap.so.1.6.2[7f6d35282000+90000]
>>> kernel: bro[4465]: segfault at 1371670 ip 00007f202ff75660 sp
>>> 00007fff04fff8c0 error 4 in libpcap.so.1.6.2[7f202ff5e000+90000]
>>> kernel: bro[4710]: segfault at 1371670 ip 00007fd8bc794660 sp
>>> 00007fff33041db0 error 4 in libpcap.so.1.6.2[7fd8bc77d000+90000]
>>> kernel: bro[7873]: segfault at 1371670 ip 00007ffc910f2660 sp
>>> 00007fff1b5ba1b0 error 4 in libpcap.so.1.6.2[7ffc910db000+90000]
>>> kernel: bro[8065]: segfault at 1371670 ip 00007ffaa5c8f660 sp
>>> 00007fff3cdde390 error 4 in libpcap.so.1.6.2[7ffaa5c78000+90000]
>>> kernel: bro[8257]: segfault at 63745e0 ip 00007ff913224660 sp
>>> 00007fff297ca2f0 error 4 in libpcap.so.1.6.2[7ff91320d000+90000]
>>> kernel: bro[8446]: segfault at 1371670 ip 00007f0a1c567660 sp
>>> 00007fffdf059910 error 4 in libpcap.so.1.6.2[7f0a1c550000+90000]
>>> kernel: bro[8638]: segfault at 1371670 ip 00007f50982af660 sp
>>> 00007fff703caa30 error 4 in libpcap.so.1.6.2[7f5098298000+90000]
>>> kernel: bro[8835]: segfault at 1371670 ip 00007f1b4acd2660 sp
>>> 00007fffacc16630 error 4 in libpcap.so.1.6.2[7f1b4acbb000+90000]
>>> kernel: bro[9036]: segfault at 1 ip 00007f10df91b506 sp 00007fff5ac3e320
>>> error 4 in libpcap.so.1.6.2[7f10df904000+90000]
>>>
>>> Regards,
>>> Gary
>>>
>>>
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>
>>
>>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro