[Bro-Dev] [JIRA] (BIT-1482) Crash from: "tcmalloc: large alloc"

Aaron Eppert (JIRA) jira at bro-tracker.atlassian.net
Tue Oct 13 16:11:00 PDT 2015


    [ https://bro-tracker.atlassian.net/browse/BIT-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=22502#comment-22502 ] 

Aaron Eppert commented on BIT-1482:
-----------------------------------

Not loading "protocols/ssl/notary" seems to "fix" the problem. This is occurring in a VM that is moderately undersized with only 4GB of RAM and two cores. I am curious if it has anything to do with "when ( local str = lookup_hostname_txt(fmt("%s.%s", digest, domain)) )" and possibly being a DNS issue?

> Crash from: "tcmalloc: large alloc"
> -----------------------------------
>
>                 Key: BIT-1482
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1482
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>            Reporter: Aaron Eppert
>         Attachments: redacted-crash-diag.log.bz2
>
>
> core.91861
> [New Thread 91861]
> [New Thread 91871]
> [New Thread 91872]
> [New Thread 91873]
> [Thread debugging using libthread_db enabled]
> Core was generated by `/usr/local/bro/bin/bro -i eth1 -U .status -p broctl -p broctl-live -p local -p'.
> Program terminated with signal 11, Segmentation fault.
> #0  0x000000000081816b in Serializer::Write (this=0x7ffde1aa2d00, v=35329, tag=0xb752df "stype") at /mnt/hgfs/src/psdev/bro/src/Serializer.h:57
> 	in /mnt/hgfs/src/psdev/bro/src/Serializer.h
> Thread 4 (Thread 0x7fb7ce219700 (LWP 91873)):
> #0  0x0000003b8f00ba0e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1  0x000000000086f551 in threading::Queue<threading::BasicInputMessage*>::Get (this=0x3e10c38) at /mnt/hgfs/src/psdev/bro/src/threading/Queue.h:173
> #2  0x000000000086dcfb in threading::MsgThread::RetrieveIn (this=0x3e10c00) at /mnt/hgfs/src/psdev/bro/src/threading/MsgThread.cc:349
> #3  0x000000000086de02 in threading::MsgThread::Run (this=0x3e10c00) at /mnt/hgfs/src/psdev/bro/src/threading/MsgThread.cc:366
> #4  0x000000000086a2c6 in threading::BasicThread::launcher (arg=0x3e10c00) at /mnt/hgfs/src/psdev/bro/src/threading/BasicThread.cc:201
> #5  0x0000003b8f007a51 in start_thread () from /lib64/libpthread.so.0
> #6  0x0000003b8ece89ad in clone () from /lib64/libc.so.6
> Thread 3 (Thread 0x7fb7cec1a700 (LWP 91872)):
> #0  0x0000003b8f00ba0e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1  0x000000000086f551 in threading::Queue<threading::BasicInputMessage*>::Get (this=0x3e11838) at /mnt/hgfs/src/psdev/bro/src/threading/Queue.h:173
> #2  0x000000000086dcfb in threading::MsgThread::RetrieveIn (this=0x3e11800) at /mnt/hgfs/src/psdev/bro/src/threading/MsgThread.cc:349
> #3  0x000000000086de02 in threading::MsgThread::Run (this=0x3e11800) at /mnt/hgfs/src/psdev/bro/src/threading/MsgThread.cc:366
> #4  0x000000000086a2c6 in threading::BasicThread::launcher (arg=0x3e11800) at /mnt/hgfs/src/psdev/bro/src/threading/BasicThread.cc:201
> #5  0x0000003b8f007a51 in start_thread () from /lib64/libpthread.so.0
> #6  0x0000003b8ece89ad in clone () from /lib64/libc.so.6
> Thread 2 (Thread 0x7fb7cf61b700 (LWP 91871)):
> #0  0x0000003b8f00ba0e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1  0x000000000086f551 in threading::Queue<threading::BasicInputMessage*>::Get (this=0x3e12438) at /mnt/hgfs/src/psdev/bro/src/threading/Queue.h:173
> #2  0x000000000086dcfb in threading::MsgThread::RetrieveIn (this=0x3e12400) at /mnt/hgfs/src/psdev/bro/src/threading/MsgThread.cc:349
> #3  0x000000000086de02 in threading::MsgThread::Run (this=0x3e12400) at /mnt/hgfs/src/psdev/bro/src/threading/MsgThread.cc:366
> #4  0x000000000086a2c6 in threading::BasicThread::launcher (arg=0x3e12400) at /mnt/hgfs/src/psdev/bro/src/threading/BasicThread.cc:201
> #5  0x0000003b8f007a51 in start_thread () from /lib64/libpthread.so.0
> #6  0x0000003b8ece89ad in clone () from /lib64/libc.so.6
> Thread 1 (Thread 0x7fb84fc06800 (LWP 91861)):
> #0  0x000000000081816b in Serializer::Write (this=0x7ffde1aa2d00, v=35329, tag=0xb752df "stype") at /mnt/hgfs/src/psdev/bro/src/Serializer.h:57
> #1  0x0000000000817fb4 in SerialObj::DoSerialize (this=0x2c2a400, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/SerialObj.cc:268
> #2  0x00000000007e1be2 in BroObj::DoSerialize (this=0x2c2a400, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Obj.cc:226
> #3  0x00000000008459b4 in BroType::DoSerialize (this=0x2c2a400, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Type.cc:283
> #4  0x000000000081788a in SerialObj::Serialize (this=0x2c2a400, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/SerialObj.cc:121
> #5  0x0000000000845670 in BroType::Serialize (this=0x2c2a400, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Type.cc:212
> #6  0x0000000000742c72 in Attributes::DoSerialize (this=0x2c2afc0, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Attr.cc:516
> #7  0x000000000081788a in SerialObj::Serialize (this=0x2c2afc0, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/SerialObj.cc:121
> #8  0x0000000000742b1b in Attributes::Serialize (this=0x2c2afc0, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Attr.cc:500
> #9  0x0000000000848ab5 in TypeDecl::Serialize (this=0x2c05ec0, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Type.cc:929
> #10 0x000000000084a01a in RecordType::DoSerialize (this=0x2aea340, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Type.cc:1250
> #11 0x000000000081788a in SerialObj::Serialize (this=0x2aea340, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/SerialObj.cc:121
> ... (pattern repeats .... )
> ...
> #116924 0x0000000000845670 in BroType::Serialize (this=0x4740480, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Type.cc:212
> #116925 0x0000000000742c72 in Attributes::DoSerialize (this=0x4808e00, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Attr.cc:516
> #116926 0x000000000081788a in SerialObj::Serialize (this=0x4808e00, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/SerialObj.cc:121
> #116927 0x0000000000742b1b in Attributes::Serialize (this=0x4808e00, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Attr.cc:500
> #116928 0x0000000000848ab5 in TypeDecl::Serialize (this=0x47eae00, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Type.cc:929
> #116929 0x000000000084a01a in RecordType::DoSerialize (this=0x4847e60, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Type.cc:1250
> #116930 0x000000000081788a in SerialObj::Serialize (this=0x4847e60, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/SerialObj.cc:121
> #116931 0x0000000000845670 in BroType::Serialize (this=0x4847e60, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Type.cc:212
> #116932 0x0000000000742c72 in Attributes::DoSerialize (this=0x48081c0, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Attr.cc:516
> #116933 0x000000000081788a in SerialObj::Serialize (this=0x48081c0, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/SerialObj.cc:121
> #116934 0x0000000000742b1b in Attributes::Serialize (this=0x48081c0, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Attr.cc:500
> #116935 0x0000000000848ab5 in TypeDecl::Serialize (this=0x47e81c0, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Type.cc:929
> #116936 0x000000000084a01a in RecordType::DoSerialize (this=0x2aec4a0, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Type.cc:1250
> #116937 0x000000000081788a in SerialObj::Serialize (this=0x2aec4a0, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/SerialObj.cc:121
> #116938 0x0000000000845670 in BroType::Serialize (this=0x2aec4a0, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Type.cc:212
> #116939 0x0000000000854a9e in Val::DoSerialize (this=0x6b92760, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Val.cc:188
> #116940 0x00000000008562bc in MutableVal::DoSerialize (this=0x6b92760, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Val.cc:656
> #116941 0x000000000085efb2 in RecordVal::DoSerialize (this=0x6b92760, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Val.cc:2813
> #116942 0x000000000081788a in SerialObj::Serialize (this=0x6b92760, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/SerialObj.cc:121
> #116943 0x0000000000854643 in Val::Serialize (this=0x6b92760, info=0x7ffde1aa2d60) at /mnt/hgfs/src/psdev/bro/src/Val.cc:100
> #116944 0x0000000000854511 in Val::Clone (this=0x6b92760) at /mnt/hgfs/src/psdev/bro/src/Val.cc:83
> #116945 0x00000000007a4d91 in Frame::Clone (this=0x8b612d0) at /mnt/hgfs/src/psdev/bro/src/Frame.cc:78
> #116946 0x0000000000841676 in Trigger::Trigger (this=0x2b79dc0, arg_cond=0x4ae81c0, arg_body=0x4af3600, arg_timeout_stmts=0x0, arg_timeout=0x0, arg_frame=0x8b612d0, arg_is_return=false, arg_location=0x4b4d280) at /mnt/hgfs/src/psdev/bro/src/Trigger.cc:108
> #116947 0x000000000083db0e in WhenStmt::Exec (this=0x4b3eba0, f=0x8b612d0, flow=@0x7ffde1aa3064) at /mnt/hgfs/src/psdev/bro/src/Stmt.cc:2166
> #116948 0x000000000083c17b in StmtList::Exec (this=0x4af4260, f=0x8b612d0, flow=@0x7ffde1aa3064) at /mnt/hgfs/src/psdev/bro/src/Stmt.cc:1764
> #116949 0x000000000083c17b in StmtList::Exec (this=0x4b56540, f=0x8b612d0, flow=@0x7ffde1aa3064) at /mnt/hgfs/src/psdev/bro/src/Stmt.cc:1764
> #116950 0x00000000007a649b in BroFunc::Call (this=0x3099030, args=0x82c33e0, parent=0x0) at /mnt/hgfs/src/psdev/bro/src/Func.cc:386
> #116951 0x000000000077f12e in EventHandler::Call (this=0x3084600, vl=0x82c33e0, no_remote=false) at /mnt/hgfs/src/psdev/bro/src/EventHandler.cc:80
> #116952 0x0000000000732965 in Event::Dispatch (this=0xb5004e0, no_remote=false) at /mnt/hgfs/src/psdev/bro/src/Event.h:50
> #116953 0x000000000077e85d in EventMgr::Dispatch (this=0xf66ee0) at /mnt/hgfs/src/psdev/bro/src/Event.cc:111
> #116954 0x000000000077e968 in EventMgr::Drain (this=0xf66ee0) at /mnt/hgfs/src/psdev/bro/src/Event.cc:128
> #116955 0x00000000007ddd66 in net_packet_dispatch (t=1442838074.400739, hdr=0x4d73140, pkt=0x7fb7db8622fc <Address 0x7fb7db8622fc out of bounds>, hdr_size=14, src_ps=0x4d73000) at /mnt/hgfs/src/psdev/bro/src/Net.cc:278
> #116956 0x0000000000af1ed6 in iosource::PktSrc::Process (this=0x4d73000) at /mnt/hgfs/src/psdev/bro/src/iosource/PktSrc.cc:411
> #116957 0x00000000007ddf6f in net_run () at /mnt/hgfs/src/psdev/bro/src/Net.cc:320
> #116958 0x00000000007319aa in main (argc=18, argv=0x7ffde1aa3af8) at /mnt/hgfs/src/psdev/bro/src/main.cc:1200
> ==== No reporter.log
> ==== stderr.log
> internal warning in /usr/local/bro/share/bro/base/frameworks/control/./main.bro, line 1: Discarded extraneous Broxygen comment: 	MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> internal warning in /usr/local/bro/share/bro/base/frameworks/control/./main.bro, line 1: Discarded extraneous Broxygen comment: 	GNU General Public License for more details.
> internal warning in /usr/local/bro/share/bro/base/frameworks/control/./main.bro, line 1: Discarded extraneous Broxygen comment: 	You should have received a copy of the GNU General Public License
> internal warning in /usr/local/bro/share/bro/base/frameworks/control/./main.bro, line 1: Discarded extraneous Broxygen comment: 	along with tcplog.  If not, see <http://www.gnu.org/licenses/>.
> listening on eth1, capture length 65535 bytes
> processing suspended
> processing continued
> tcmalloc: large alloc 1562509312 bytes == 0x498f0000 @  0x7fb85004b4ac 0x7fb85006b22c 0x73b0e5 0x815270 0x81627e 0x7437f8 0x742ddd 0x81788a 0x742b1b 0x848ab5 0x84a01a 0x81788a 0x845670 0x848b3b 0x84a01a 0x81788a 0x845670 0x846db0 0x84759e 0x81788a 0x845670 0x742c72 0x81788a 0x742b1b 0x848ab5 0x84a01a 0x81788a 0x845670 0x742c72 0x81788a 0x742b1b
> /usr/local/bro/share/broctl/scripts/run-bro: line 85: 91861 Segmentation fault      (core dumped) nohup ${pin_command} $pin_cpu $mybro "$@"
> ----
> (gdb) frame 0
> #0  0x000000000081816b in Serializer::Write (this=0x7ffde1aa2d00, v=35329, tag=0xb752df "stype")
>     at /mnt/hgfs/src/psdev/bro/src/Serializer.h:57
> 57		DECLARE_IO(uint16)
> (gdb) print *this
> $8 = {_vptr.Serializer = 0xb7dc50, static MAGIC = 1112691540, static DATA_FORMAT_VERSION = 25, io = 0x0, format = 0x89def00, 
>   current_cache = 0x0, error_descr = 0x0}
> (gdb) print *this
> $10 = {_vptr.Serializer = 0xb7dc50, static MAGIC = 1112691540, static DATA_FORMAT_VERSION = 25, io = 0x0, 
>   format = 0x89def00, current_cache = 0x0, error_descr = 0x0}
> (gdb) print *this->format
> $11 = {_vptr.SerializationFormat = 0xb74dd0, static INITIAL_SIZE = 65536, static GROWTH_FACTOR = 2.5, 
>   output = 0x498f0000 "\001", output_size = 1562499968, output_pos = 852829181, input = 0x0, input_len = 0, input_pos = 0, 
>   bytes_written = 852829181, bytes_read = 0}
> The stack trace and the problem seems to be similar to:
> http://mailman.icsi.berkeley.edu/pipermail/bro/2015-March/008241.html



--
This message was sent by Atlassian JIRA
(v7.0.0-OD-08-001#70107)


More information about the bro-dev mailing list