<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-GB">
<div class="WordSection1">
<p class="MsoNormal">Hi Everyone,</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I'm using the "find_all_urls()" function from urls.zeek to extract all URLs from HTTP bodies. I occasionally errors such as this these:</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">1485557634.826679 error in /usr/local/zeek/share/zeek/base/utils/urls.zeek, line 122: bad conversion to count (to_count(parts[1]) and answers:PersonalBing:EZBubbleClose) no-repeat center;width:11px;height:11px;background-position-y:-10px}#hp_bottomCell
#ezp_notification #ezp_bubble .ezp_bubble_close:hover{background-position-y:0}.ezp_location{font:14px)</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">1485557634.826679 error in /usr/local/zeek/share/zeek/base/utils/urls.zeek, line 122: bad conversion to count (to_count(parts[1]) and answers:PersonalBing:EZPanelClose) no-repeat center;width:11px;height:11px}.ezp_module{float:left;height:269px;width:255px;margin:25px
0;padding:0 42px}.ezp_module.ezp_module_narrow{width:122px}.ezp_module_leftseparator{border-left:1px solid #222}.ezp_module_title{font-size:20px;line-height:24px;margin-bottom:11px}.ezp_module_desc{font-size:16px;line-height:20px;margin-bottom:20px}.ezp_interests_icon{vertical-align:middle}.ezp_option_control{background:url(rms:)</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">1485557634.826679 error in /usr/local/zeek/share/zeek/base/utils/urls.zeek, line 122: bad conversion to count (to_count(parts[1]) and answers:PersonalBing:EZPanelClose) no-repeat center;width:11px;height:11px;position:relative;top:-22px;left:-10px}#hp_tbar.ezp_signin_message{background-image:-webkit-gradient(linear,left
top,left bottom,from(rgba(0,0,0,.55)),to(rgba(0,0,0,.85)));background-image:-moz-linear-gradient(rgba(0,0,0,.55) 0,rgba(0,0,0,.85) 80%);background-image:-ms-linear-gradient(rgba(0,0,0,.55) 0,rgba(0,0,0,.85) 80%);background-image:-o-linear-gradient(rgba(0,0,0,.55)
0,rgba(0,0,0,.85) 80%);background-image:linear-gradient(rgba(0,0,0,.55) 0,rgba(0,0,0,.85) 80%)}.ezp_opened .ezp_barrier{display:block;background-color:#000;height:111px;margin:0 40px;position:relative;top:-185px;opacity:0}#sc_mdc.loading+.ezp_panelopened{margin-top:-46px}.ezp_icon{position:relative;top:-5px;left:0;cursor:pointer;background-color:rgba(34,34,34,.75);margin-right:1px;margin-bottom:-7px;-webkit-margin-after:-5px}#ezp_bubble_message{position:absolute;left:30px;background-color:rgba(0,0,0,.8);color:#fff;border:1px
solid #333;padding:0 12px;font-size:13px;line-height:40px;height:40px;opacity:0}#ezp_bubble_message .ezp_info{vertical-align:middle;margin-right:12px}#ezp_bubble_message .ezp_bubble_down{background:url(rms:)</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">1378597102.912603 error in /usr/local/zeek/share/zeek/base/utils/urls.zeek, line 122: bad conversion to count (to_count(parts[1]) and )</p>
<p class="MsoNormal">www.iec.ch\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x16IEC http</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have a couple of questions regarding this:</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">1) When trying to resolve some of these issues, should I directly modify urls.zeek or will this have unintended consequences regarding other scripts/functionality in Zeek? The reason I ask this is when printing URLs extracted with the find_all_urls()
function I get some results which are clearly not valid URLs e.g. "http://www.yootheme.com/license) */" - this should have cut off before the ")" which I believe are bug with urls.zeek rather than simply being intended functionality that I'd like to change.</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">2) Assuming I don't manage to fix all of these errors and choose to accept some, how can I stop them from printing to console each time I process a PCAP?</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">3) While trying to fix some of these errors with regex, I ran into the example "www.iec.ch\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x16IEC http". I've tried to strip everthing after the first "\" but this doesn't work due to it being
Hex (I guess) rather than an actual "\", any ideas for this specific case?</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">4) Finally, a regex related question I've been meaning to ask for a while. Because I'm trying to extract URLs from HTML/JS, I need to deal with cases whitespace and multiple types of quote character may be used. When I've written projects
in Python, I would create a variable with all of the possible characters in it and then I would use this variable in the regex e.g.</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">q = r"[\‘\’\'\"\s]*(?:&quot|&#039;)*"</p>
<p class="MsoNormal">pattern = q+r"userTokens"+q+r"(?::|=)"+q+r"(\w+)"+q</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">if re.search(pattern, data):</p>
<p class="MsoNormal"> do something..</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">I can't workout how to do this with regex in Bro/Zeek scripts so I'm having to create incredibly long patterns to ensure all possible cases are met, if anybody can recommend a better way (like how I did it in Python), that would be awesome!</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks in Advance,</p>
<p class="MsoNormal">Jonah (CryptoCat)</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>