[Bro-Dev] [JIRA] (BIT-1289) Ascii Stream Reader not freeing up memory

[Johanna Amann (JIRA)]

     [ https://bro-tracker.atlassian.net/browse/BIT-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Johanna Amann reassigned BIT-1289:
----------------------------------

    Assignee: Johanna Amann

> Ascii Stream Reader not freeing up memory 
> ------------------------------------------
>
>                 Key: BIT-1289
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1289
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: 2.2, 2.3
>         Environment: CentOS 6.6 distro x86_64
> gcc 4.4.7 20120313 (Red Hat 4.4.7-4)
> ld 2.20.51.0.2-5.36.el6 20100205
> glibc 2.12 1.32.el6_5.4
> libunwind 1.1 2.el6
> libstdc++ 4.4.7 4.el6
> libm (via glibc-devel 2.12 1.07.el6_4.4)
>            Reporter: Steve Egbert
>            Assignee: Johanna Amann
>              Labels: input-framework
>         Attachments: bro-ascii-stream-event-leak.patch
>
>
> A triple leak occurred whenever an ASCII file has its modify time changed {{DoUpdate}} as well as once during {{DoInit}} initialization which called {{DoUpdate}}. 
> When tracing the EVENT_STREAM message from reading of an input file using Ascii framework, but I spotted a memory leak that impacts both Bro 2.2 (and possibly 2.3) as reported by gperftool/tcmalloc {{HEAPCHECK}} output:
>   Using local file /usr/bin/bro.
>   Leak of 5148576 bytes in 160893 objects allocated from:
>         @ 7ab2b5 AsciiFormatter::ParseValue
>         @ 7d9404 input::reader::Ascii::DoUpdate
>         @ 7d665c input::ReaderBackend::Update
>         @ 7d97ba input::reader::Ascii::DoHeartbeat
>         @ 7d674a input::ReaderBackend::OnHeartbeat
>         @ 7b0e95 threading::HeartbeatMessage::Process
>         @ 7b085a threading::MsgThread::Run
>         @ 7ad264 threading::BasicThread::launcher
>         @ 37340079d1 start_thread
>         @ 37338e886d __clone
>         @ 0 _init
>   Leak of 5148480 bytes in 160890 objects allocated from:
>         @ 7d91e2 input::reader::Ascii::DoUpdate
>         @ 7d665c input::ReaderBackend::Update
>         @ 7d97ba input::reader::Ascii::DoHeartbeat
>         @ 7d674a input::ReaderBackend::OnHeartbeat
>         @ 7b0e95 threading::HeartbeatMessage::Process
>         @ 7b085a threading::MsgThread::Run
>         @ 7ad264 threading::BasicThread::launcher
>         @ 37340079d1 start_thread
>         @ 37338e886d __clone
>         @ 0 _init
>   Leak of 2062752 bytes in 160854 objects allocated from:
>         @ 673735 copy_string
>         @ 7ab399 AsciiFormatter::ParseValue
>         @ 7d9404 input::reader::Ascii::DoUpdate
>         @ 7d665c input::ReaderBackend::Update
>         @ 7d97ba input::reader::Ascii::DoHeartbeat
>         @ 7d674a input::ReaderBackend::OnHeartbeat
>         @ 7b0e95 threading::HeartbeatMessage::Process
>         @ 7b085a threading::MsgThread::Run
>         @ 7ad264 threading::BasicThread::launcher
>         @ 37340079d1 start_thread
>         @ 37338e886d __clone
>         @ 0 _init
> In short, the leakage was found to occur in the Manager during processing incoming EVENT_STREAM messages that was sent by its child thread(s) via {{queue_out}} queue.  Manager used {{RetrieveOut()}} of getting these messages from the {{queue_out}} and eventually calling {{SendEntry()}} for final processing and disposal of these messages (note the {{delete_vals_ptr_array()}} at the end of the {{SendEntry()}} function).  But the error was in miscomputing of {[readFields}} for its actual number of fields associating with the msg/vals/fields.
> In {{SendEntry()}} (of {{Manager.cc}}), note the out-of-sync between EVENT_STREAM's {{readField}} assignment and the original {{stream->num_fields}} waaaay back in when the message was first created by the {{DoInit}}.
> So, when it comes to counting number of columns/fields, it becomes an issue of whether to go by what was in the first line of the text file describing the columns or what was given in the data line afterwards.
> Perhaps a better value is to use {{(EventStream *)i->num_fields}} instead, but only for EVENT_STREAM?  I've demonstrated removal of these leaks using this field instead.
> We've already lost the {{columnMap}} details at file read time which would have made for a better comparator value.



--
This message was sent by Atlassian JIRA
(v6.4-OD-09-008#64005)