[Bro-Dev] Making scan.bro great again.

[Azoff, Justin S]

I know.. I send too many emails :-)

I let the rewritten script run over the weekend, cpu and memory was stable.

I added one additional table to store known scanners so it can completely purge a scanners state, this cut down on the total amount of data stored by 1/2, as measured by

    while true;do echo $(date) $(broctl print Scan::recent_scan_attempts |sort -u| wc -l);sleep 30m;done | tee -a keys.log

currently this is around 155,000 for us. That is 155,000 addr, port records.  approx 16 bytes for each ip and 2 bytes for each port, gives ~3 MB of raw data, times whatever the overhead in bro is.

I also fixed the duration and port formatting issues, so now it properly shows things like


   ... scanned at least 100 unique hosts on port 3306/tcp in 13m18s
   ... scanned at least 70 unique hosts on ports 23/tcp, 2222/tcp, 22/tcp in 102m27s
   ... scanned at least 100 unique hosts on port 23/tcp in 0m1s
   ... scanned at least 99 hosts on 80 ports in 0m52s

I also even further simplified the connection filtering that feeds into scan detecting, I think it now finally has the bare minimum needed to detect scans and does not flag connections with capture loss as scans.


The last graph I included was a bit of a mess, this one is a little more clear


[cid:fd51c96b-a455-49b6-93ef-13f31ce0325c at mx.uillinois.edu]



It shows 3 experiments, from left to right:

* Stock scan.bro
* Unified scan.bro that still uses sumstats
* Unified scan.bro rewritten to not use sumstats and to work like the 1.5 version did (attached)

Also interesting is a graph of the network traffic during the same timeframe:

[cid:d97f3aab-1be2-466d-965c-d8a1e57a127e at mx.uillinois.edu]


The positive line is manager -> worker traffic, and the negative line is worker -> manager traffic.

The negative line includes log writes, so the floor there won't be zero.



--
- Justin Azoff
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160801/e4495fef/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bro-test-manager-cpu.png
Type: image/png
Size: 55644 bytes
Desc: bro-test-manager-cpu.png
Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160801/e4495fef/attachment-0002.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bro-test-network.png
Type: image/png
Size: 53849 bytes
Desc: bro-test-network.png
Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160801/e4495fef/attachment-0003.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: scan.bro
Type: application/octet-stream
Size: 6695 bytes
Desc: scan.bro
Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160801/e4495fef/attachment-0001.obj