[Bro] Long lived MySQL session stops generating events

Bruce Kao brucekao at heliosdata.com
Mon Oct 15 13:37:34 PDT 2018


Thanks Jon for the tips.  I did see that Wireshark is reporting malformed response data, but since Bro was able to produce events initially I did not dig into it too much.  My MySQL was installed through Ubuntu which is


mysqld  Ver 5.7.23-0ubuntu0.16.04.1 for Linux on x86_64 ((Ubuntu))


I will follow your tips and see if I can find more information and report back.


Thanks

________________________________
From: Jon Siwek <jsiwek at corelight.com>
Sent: Monday, October 15, 2018 12:08:59 PM
To: Bruce Kao
Cc: bro
Subject: Re: [Bro] Long lived MySQL session stops generating events

You can get a bit more in to the mysql.log if you ran like:

  bro -r out.pcap tcp_inactivity_timeout=10min
"DPD::ignore_violations+={Analyzer::ANALYZER_MYSQL}"

There were a couple potential issues that's working around when I
tested with Bro's git/master branch.

The first was that the responses may be malformed -- Wireshark appears
to report that it's malformed MySQL and Bro also indicates a parsing
problem in dpd.log, showing that the LengthEncodedIntegerLookahead
field was expecting to read 8 bytes, but only 6 were available/sent.
The default behavior is for Bro to disable analyzers that encounter
such protocol violations, but the DPD::ignore_violations customization
prevents that.  Someone may ultimately want to check into whether the
packet is legit following the MySQL protocol, because then the parser
in Bro should be updated to understand it better.

The second issue was related to the long idle period that you
mentioned.  The default timeout for TCP sessions in Bro is 5mins, so
when eventually new MySQL data is seen, it instantiates a new
session/analyzer, however, the MySQL analyzer currently does not
handle "partial connections" and everything from then on is just
skipped.  You can look into tuning the tcp_inactivity_timeout setting
to potentially workaround that, or else it would be better if anyone
wanted to investigate how much it would take to get the MySQL
parser/analyzer able to pick up and analyze sessions already mid-way
in progress.

- Jon

On Mon, Oct 15, 2018 at 12:07 PM Bruce Kao <brucekao at heliosdata.com> wrote:
>
> Hi Jon,
>
> I tried the same scenario with Bro 2.5.5 and the same problem with MySQL long session is still reproducible.  I am attaching the pcap here which should show the same issue.
>
>
> It is a straight forward MySQL session using mysql command.  There is a long period of idling between packet 23 and packet 24, and no mysql events can be generated after packet 24.
>
>
> The last working command with MySQL event generated is packet 21.
>
>
> Any suggestions on how to debug this issue would be greatly appreciated.  I am not very familiar with how Bro passes received data into the analyzers for the message parsing, but I would like to find out whether it is that the parsing is failing or if Bro is not picking up the data at all.
>
>
> Thanks in advance for any guidance anyone can provide.
>
> ________________________________
> From: Jon Siwek <jsiwek at corelight.com>
> Sent: Friday, October 12, 2018 3:03:55 PM
> To: Bruce Kao
> Cc: bro
> Subject: Re: [Bro] Long lived MySQL session stops generating events
>
> On Thu, Oct 11, 2018 at 7:52 PM Bruce Kao <brucekao at heliosdata.com> wrote:
>
> > I am running Bro version 2.5.2, and encountering an issue where an established MySQL session after idling for some time (approximately 10 minutes), the MySQL analyzer events are no longer generated.
>
> I'd first suggest trying with the latest release of Bro (2.5.5) if you
> can because the MySQL analyzer had some general changes/fixes in
> 2.5.4.
>
> Otherwise, it would be most helpful if you could provide a pcap that
> reproduces the issue.
>
> - Jon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181015/b8d21cf5/attachment.html 


More information about the Bro mailing list