[Bro] bro elasticsearch plugin + kibana indexing

Fri Mar 10 00:54:04 PST 2017

Google for it :D

> On 08 Mar 2017, at 21:42, Alex Kefallonitis <al.kefallonitis at gmail.com> wrote:
> 
> How TS_ISO8601 timestamp looks like? I ll try to recompile maybe i didn't apply the patch correctly. Thanks again a lot for your help
> 
> 2017-03-05 15:41 GMT+02:00 Daniel Guerra <daniel.guerra69 at gmail.com <mailto:daniel.guerra69 at gmail.com>>:
> The patch wasn’t used, your timestamp is not in TS_ISO8601 but in TS_MILIS
> And your separator is a “.” not a “_"
> 
> Check my docker 
> 
> https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/ <https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/>
> 
>> On 05 Mar 2017, at 14:12, Alex Kefallonitis <al.kefallonitis at gmail.com <mailto:al.kefallonitis at gmail.com>> wrote:
>> 
>> Nothing changed
>> 
>> <image.png>
>> 
>> 
>> Although i have logs
>> 
>> tail -f /opt/bro/logs/current/conn.log 
>> {"ts":1488719244.873684,"uid":"CCToVE1JzVl9n5zDnj","id.orig_h":"10.0.0.31","id.orig_p":123,"id.resp_h":"194.177.210.54","id.resp_p":123,"proto":"udp","duration":0.021199,"orig_bytes":0,"resp_bytes":48,"conn_state":"SHR","local_orig":true,"local_resp":false,"missed_bytes":0,"history":"Cd","orig_pkts":0,"orig_ip_bytes":0,"resp_pkts":1,"resp_ip_bytes":76,"tunnel_parents":[],"orig_l2_addr":"b8:27:eb:68:1a:49","resp_l2_addr":"36:34:64:31:64:39"}
>> {"ts":1488719252.873686,"uid":"ChHXxg3NsigjS6QwXg","id.orig_h":"10.0.0.31","id.orig_p":123,"id.resp_h":"62.1.45.120","id.resp_p":123,"proto":"udp","duration":0.020193,"orig_bytes":0,"resp_bytes":48,"conn_state":"SHR","local_orig":true,"local_resp":false,"missed_bytes":0,"history":"Cd","orig_pkts":0,"orig_ip_bytes":0,"resp_pkts":1,"resp_ip_bytes":76,"tunnel_parents":[],"orig_l2_addr":"b8:27:eb:68:1a:49","resp_l2_addr":"36:34:64:31:64:39"}
>> {"ts":1488719249.686949,"uid":"CIrHst2VsHafEIR4vk","id.orig_h":"10.0.0.3","id.orig_p":123,"id.resp_h":"91.189.89.198","id.resp_p":123,"proto":"udp","duration":0.066331,"orig_bytes":48,"resp_bytes":48,"conn_state":"SF","local_orig":true,"local_resp":false,"missed_bytes":0,"history":"Dd","orig_pkts":1,"orig_ip_bytes":76,"resp_pkts":1,"resp_ip_bytes":76,"tunnel_parents":[],"orig_l2_addr":"32:38:66:64:64:62","resp_l2_addr":"36:34:64:31:64:39"}
>> {"ts":1488719306.835847,"uid":"CqrMPYWEW543RxOX5","id.orig_h":"10.0.0.33","id.orig_p":51666,"id.resp_h":"10.0.0.1","id.resp_p":53,"proto":"udp","service":"dns","duration":0.000744,"orig_bytes":39,"resp_bytes":98,"conn_state":"SF","local_orig":true,"local_resp":true,"missed_bytes":0,"history":"Dd","orig_pkts":1,"orig_ip_bytes":67,"resp_pkts":1,"resp_ip_bytes":126,"tunnel_parents":[],"orig_l2_addr":"36:33:62:63:39:61","resp_l2_addr":"36:34:64:31:64:39"}
>> {"ts":1488719314.06168,"uid":"CYw1dj2WUBXn6ua8O1","id.orig_h":"10.0.0.31","id.orig_p":37456,"id.resp_h":"10.0.0.1","id.resp_p":53,"proto":"udp","service":"dns","duration":0.000596,"orig_bytes":0,"resp_bytes":172,"conn_state":"SHR","local_orig":true,"local_resp":true,"missed_bytes":0,"history":"Cd","orig_pkts":0,"orig_ip_bytes":0,"resp_pkts":2,"resp_ip_bytes":228,"tunnel_parents":[],"orig_l2_addr":"b8:27:eb:68:1a:49","resp_l2_addr":"36:34:64:31:64:39"}
>> 
>> Do i have to change serator to init.bro also?
>> 
>> 
>> 2017-03-05 14:57 GMT+02:00 Daniel Guerra <daniel.guerra69 at gmail.com <mailto:daniel.guerra69 at gmail.com>>:
>> ##! Load this script to enable global log output to an ElasticSearch database.
>> 
>> module LogElasticSearch;
>> 
>> export {
>> 	## An elasticsearch specific rotation interval.
>> 	const rotation_interval = 1hr &redef;
>> 
>> 	## Optionally ignore any :bro:type:`Log::ID` from being sent to
>> 	## ElasticSearch with this script.
>> 	const excluded_log_ids: set[Log::ID] &redef;
>> 
>> 	## If you want to explicitly only send certain :bro:type:`Log::ID`
>> 	## streams, add them to this set.  If the set remains empty, all will
>> 	## be sent.  The :bro:id:`LogElasticSearch::excluded_log_ids` option
>> 	## will remain in effect as well.
>> 	const send_logs: set[Log::ID] &redef;
>> 
>> 	## Set the separator
>> 	redef Log::default_scope_sep = "_";
>> }
>> 
>> 
>>> On 05 Mar 2017, at 13:56, Alex Kefallonitis <al.kefallonitis at gmail.com <mailto:al.kefallonitis at gmail.com>> wrote:
>>> 
>>> Where do i put this? 
>>> 
>>> redef Log::default_scope_sep = "_”;
>>> 
>>> Do i have to enable json output to ascii.bro?
>>> 
>>> 2017-03-05 14:32 GMT+02:00 Daniel Guerra <daniel.guerra69 at gmail.com <mailto:daniel.guerra69 at gmail.com>>:
>>> Don’t forget this in the bro script that starts elasticsearch in the export part
>>> 
>>> redef Log::default_scope_sep = "_”;
>>> 
>>> 
>>>> On 05 Mar 2017, at 11:22, Alex Kefallonitis <al.kefallonitis at gmail.com <mailto:al.kefallonitis at gmail.com>> wrote:
>>>> 
>>>> I do patch src/ElasticSearch.cc <http://elasticsearch.cc/> ./ElasticSearch.cc.patch ./configure && make && make install . Load bro elasticsearch script and restart bro open kibana 
>>>> 
>>>> <image.png>
>>>> 
>>>> 2017-03-05 12:14 GMT+02:00 Alex Kefallonitis <al.kefallonitis at gmail.com <mailto:al.kefallonitis at gmail.com>>:
>>>> I try the patch too but still no timestamp appears i am using ELK 5.2.2
>>>> 
>>>> 2017-03-05 10:27 GMT+02:00 Daniel Guerra <daniel.guerra69 at gmail.com <mailto:daniel.guerra69 at gmail.com>>:
>>>> Try this
>>>> 
>>>> https://github.com/danielguerra69/bro-debian-elasticsearch/blob/master/bro-patch/ElasticSearch.cc.patch <https://github.com/danielguerra69/bro-debian-elasticsearch/blob/master/bro-patch/ElasticSearch.cc.patch>
>>>> 
>>>> 
>>>> > On 05 Mar 2017, at 02:57, Alex Kefallonitis <al.kefallonitis at gmail.com <mailto:al.kefallonitis at gmail.com>> wrote:
>>>> >
>>>> > ELK + Kibana not indexing bro logs
>>>> >
>>>> > Succesfully installed the plugin and ELK but when i add indexing bro-* , index time-field appears empty (@timestamp) so i cannot use bro logs with kibana search. Anyone have same issue?
>>>> > _______________________________________________
>>>> > Bro mailing list
>>>> > bro at bro-ids.org <mailto:bro at bro-ids.org>
>>>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170310/af813506/attachment-0001.html