<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body>
<div class="BodyFragment"><font size="2"><span style="font-size:10pt;">
<div class="PlainText"><br>
> On May 20, 2016, at 4:52 PM, Seth Hall <seth@icir.org> wrote:<br>
> <br>
> For the 2.5 release, we were hoping to understand why the topic/seth/remove-flare fixes some issues that people have been seeing with the communication code. Perhaps even more to the point we are aiming to understand why that branch fixes the problem, but
Robin's branch topic/robin/no-flares-2.4.1 doesn't work. <br>
> <br>
> The problem that we've seen will exhibit on Linux (for some reason FreeBSD doesn't seem to be affected) and you will see high memory use on the child of your manager process. People will tend to notice it in two ways.<br>
> 1. Memory exhaustion<br>
> 2. Logs being written that are seconds to minutes old.<br>
> <br>
> This isn't exactly a request for anyone to do anything, but more a call for anyone that would like to dig around in the core to figure out what is going on here so we can get a fix merged into master.<br>
> <br>
> Thanks!<br>
> .Seth<br>
<br>
I had looked into it a while ago.. I don't think the differences in your branches has anything to do with flares...<br>
<br>
$ git diff origin/topic/robin/no-flares-2.4.1 origin/topic/seth/remove-flare src/iosource/Manager.cc<br>
diff --git a/src/iosource/Manager.cc b/src/iosource/Manager.cc<br>
index 80fa5fe..5ad8cca 100644<br>
--- a/src/iosource/Manager.cc<br>
+++ b/src/iosource/Manager.cc<br>
@@ -96,8 +96,8 @@ IOSource* Manager::FindSoonest(double* ts)<br>
// return it.<br>
int maxx = 0;<br>
<br>
- if ( soonest_src && (call_count % SELECT_FREQUENCY) != 0 )<br>
- goto finished;<br>
+// if ( soonest_src && (call_count % SELECT_FREQUENCY) != 0 )<br>
+// goto finished;<br>
<br>
// Select on the join of all file descriptors.<br>
fd_set fd_read, fd_write, fd_except;<br>
<br>
<br>
$ git diff origin/topic/robin/no-flares-2.4.1 origin/topic/seth/remove-flare src/RemoteSerializer.cc<br>
[snip]<br>
<br>
- // FIXME: Fine-tune this (timeouts, flush, etc.)<br>
- struct timeval small_timeout;<br>
- small_timeout.tv_sec = 0;<br>
- small_timeout.tv_usec =<br>
- io->CanWrite() || io->CanRead() ? 1 : 10;<br>
-<br>
-#if 0<br>
- if ( ! io->CanWrite() )<br>
- usleep(10);<br>
-#endif<br>
-<br>
- int a = select(max_fd + 1, &fd_read, &fd_write, &fd_except,<br>
- &small_timeout);<br>
+ struct timeval timeout;<br>
+ timeout.tv_sec = 1;<br>
+ timeout.tv_usec = 0;<br>
<br>
- if ( a == 0 )<br>
- ++timeouts;<br>
+ int a = select(max_fd + 1, &fd_read, &fd_write, &fd_except, &timeout);<br>
<br>
<br>
Seths branch removes the SELECT_FREQUENCY check and defaults the serializer 'small timeout' to 1 full second. Robins branch still has the SELECT_FREQUENCY check and has the small timeout set to 1 or 10 microseconds. I think the two extra changes in Seths
branch combine to make bro spend more time in the RemoteSerializer code.<br>
<br>
When I was trying to figure some of this out I believed that many of these constants were part of the issue. All the different places calling select with different timeouts and different frequencies causing bro to spend more time calling select than it was
actually moving bytes around. The only thing I ever really found wrong with the flare code was that repeated fire/extinguishes were not No-Ops and I had a small patch that improved that without changing anything else (attached).<br>
<br>
I think Robins branch doesn't fix the problem because I don't think the flares were really the issue.. I think bro started having issues because between 2.3 and 2.4 traffic volumes increased, cluster sizes increased, and we added a ton of new analyzers and
log files which put even more strain on the communication system.<br>
<br>
<br>
<br>
<br>
</div>
</span></font></div>
<div class="BodyFragment"><font size="2"><span style="font-size:10pt;">
<div class="PlainText"><br>
<br>
-- <br>
- Justin Azoff<br>
<br>
</div>
</span></font></div>
</body>
</html>