[Bro-Dev] [JIRA] (BIT-1289) Ascii Stream Reader not freeing up memory
Steve Egbert (JIRA)
jira at bro-tracker.atlassian.net
Wed Nov 19 09:21:07 PST 2014
[ https://bro-tracker.atlassian.net/browse/BIT-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Egbert updated BIT-1289:
------------------------------
Attachment: bro-ascii-stream-event-leak.patch
> Ascii Stream Reader not freeing up memory
> ------------------------------------------
>
> Key: BIT-1289
> URL: https://bro-tracker.atlassian.net/browse/BIT-1289
> Project: Bro Issue Tracker
> Issue Type: Problem
> Components: Bro
> Affects Versions: 2.2, 2.3
> Environment: CentOS 6.6 distro x86_64
> gcc 4.4.7 20120313 (Red Hat 4.4.7-4)
> ld 2.20.51.0.2-5.36.el6 20100205
> glibc 2.12 1.32.el6_5.4
> libunwind 1.1 2.el6
> libstdc++ 4.4.7 4.el6
> libm (via glibc-devel 2.12 1.07.el6_4.4)
> Reporter: Steve Egbert
> Labels: input-framework
> Attachments: bro-ascii-stream-event-leak.patch
>
>
> A triple leak occurred whenever an ASCII file has its modify time changed {{DoUpdate}} as well as once during {{DoInit}} initialization which called {{DoUpdate}}.
> When tracing the EVENT_STREAM message from reading of an input file using Ascii framework, but I spotted a memory leak that impacts both Bro 2.2 (and possibly 2.3) as reported by gperftool/tcmalloc {{HEAPCHECK}} output:
> Using local file /usr/bin/bro.
> Leak of 5148576 bytes in 160893 objects allocated from:
> @ 7ab2b5 AsciiFormatter::ParseValue
> @ 7d9404 input::reader::Ascii::DoUpdate
> @ 7d665c input::ReaderBackend::Update
> @ 7d97ba input::reader::Ascii::DoHeartbeat
> @ 7d674a input::ReaderBackend::OnHeartbeat
> @ 7b0e95 threading::HeartbeatMessage::Process
> @ 7b085a threading::MsgThread::Run
> @ 7ad264 threading::BasicThread::launcher
> @ 37340079d1 start_thread
> @ 37338e886d __clone
> @ 0 _init
> Leak of 5148480 bytes in 160890 objects allocated from:
> @ 7d91e2 input::reader::Ascii::DoUpdate
> @ 7d665c input::ReaderBackend::Update
> @ 7d97ba input::reader::Ascii::DoHeartbeat
> @ 7d674a input::ReaderBackend::OnHeartbeat
> @ 7b0e95 threading::HeartbeatMessage::Process
> @ 7b085a threading::MsgThread::Run
> @ 7ad264 threading::BasicThread::launcher
> @ 37340079d1 start_thread
> @ 37338e886d __clone
> @ 0 _init
> Leak of 2062752 bytes in 160854 objects allocated from:
> @ 673735 copy_string
> @ 7ab399 AsciiFormatter::ParseValue
> @ 7d9404 input::reader::Ascii::DoUpdate
> @ 7d665c input::ReaderBackend::Update
> @ 7d97ba input::reader::Ascii::DoHeartbeat
> @ 7d674a input::ReaderBackend::OnHeartbeat
> @ 7b0e95 threading::HeartbeatMessage::Process
> @ 7b085a threading::MsgThread::Run
> @ 7ad264 threading::BasicThread::launcher
> @ 37340079d1 start_thread
> @ 37338e886d __clone
> @ 0 _init
> In short, the leakage was found to occur in the Manager during processing incoming EVENT_STREAM messages that was sent by its child thread(s) via {{queue_out}} queue. Manager used {{RetrieveOut()}} of getting these messages from the {{queue_out}} and eventually calling {{SendEntry()}} for final processing and disposal of these messages (note the {{delete_vals_ptr_array()}} at the end of the {{SendEntry()}} function). But the error was in miscomputing of {[readFields}} for its actual number of fields associating with the msg/vals/fields.
> In {{SendEntry()}} (of {{Manager.cc}}), note the out-of-sync between EVENT_STREAM's {{readField}} assignment and the original {{stream->num_fields}} waaaay back in when the message was first created by the {{DoInit}}.
> So, when it comes to counting number of columns/fields, it becomes an issue of whether to go by what was in the first line of the text file describing the columns or what was given in the data line afterwards.
> Perhaps a better value is to use {{(EventStream *)i->num_fields}} instead, but only for EVENT_STREAM? I've demonstrated removal of these leaks using this field instead.
> We've already lost the {{columnMap}} details at file read time which would have made for a better comparator value.
--
This message was sent by Atlassian JIRA
(v6.4-OD-09-008#64005)
More information about the bro-dev
mailing list