<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body>
<div class="BodyFragment"><font size="2"><span style="font-size:10pt;">
<div class="PlainText">I know.. I send too many emails :-)<br>
<br>
I let the rewritten script run over the weekend, cpu and memory was stable.<br>
<br>
I added one additional table to store known scanners so it can completely purge a scanners state, this cut down on the total amount of data stored by 1/2, as measured by<br>
<br>
while true;do echo $(date) $(broctl print Scan::recent_scan_attempts |sort -u| wc -l);sleep 30m;done | tee -a keys.log<br>
<br>
currently this is around 155,000 for us. That is 155,000 addr, port records. approx 16 bytes for each ip and 2 bytes for each port, gives ~3 MB of raw data, times whatever the overhead in bro is.<br>
<br>
I also fixed the duration and port formatting issues, so now it properly shows things like<br>
<br>
<br>
... scanned at least 100 unique hosts on port 3306/tcp in 13m18s<br>
... scanned at least 70 unique hosts on ports 23/tcp, 2222/tcp, 22/tcp in 102m27s<br>
... scanned at least 100 unique hosts on port 23/tcp in 0m1s<br>
... scanned at least 99 hosts on 80 ports in 0m52s<br>
<br>
I also even further simplified the connection filtering that feeds into scan detecting, I think it now finally has the bare minimum needed to detect scans and does not flag connections with capture loss as scans.<br>
<br>
<br>
The last graph I included was a bit of a mess, this one is a little more clear<br>
<br>
<br>
</div>
</span></font></div>
<div><img src="cid:fd51c96b-a455-49b6-93ef-13f31ce0325c@mx.uillinois.edu"> </div>
<div class="BodyFragment"><font size="2"><span style="font-size:10pt;">
<div class="PlainText"><br>
<br>
<br>
It shows 3 experiments, from left to right:<br>
<br>
* Stock scan.bro<br>
* Unified scan.bro that still uses sumstats<br>
* Unified scan.bro rewritten to not use sumstats and to work like the 1.5 version did (attached)<br>
<br>
Also interesting is a graph of the network traffic during the same timeframe:<br>
<br>
</div>
</span></font></div>
<div><img src="cid:d97f3aab-1be2-466d-965c-d8a1e57a127e@mx.uillinois.edu"> </div>
<div class="BodyFragment"><font size="2"><span style="font-size:10pt;">
<div class="PlainText"><br>
<br>
The positive line is manager -> worker traffic, and the negative line is worker -> manager traffic.<br>
<br>
The negative line includes log writes, so the floor there won't be zero.<br>
<br>
</div>
</span></font></div>
<div class="BodyFragment"><font size="2"><span style="font-size:10pt;">
<div class="PlainText"><br>
<br>
-- <br>
- Justin Azoff</div>
</span></font></div>
</body>
</html>