[Bro-Dev] [JIRA] (BIT-1504) The facility to serialize tables to a log

[Aaron Eppert (JIRA)]

    [ https://bro-tracker.atlassian.net/browse/BIT-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=22810#comment-22810 ] 

Aaron Eppert commented on BIT-1504:
-----------------------------------

First, thank you very much Johanna for your reply and the detailed information. After digging for awhile and testing a few theories, I realized it was a design issue and was likely around the original ascii conventions of Bro, but it is always very helpful to know the details from someone far more familiar with the code base.

I do stand by the original feature request given the complexity of protocols aren't decreasing and the importance of various metadata for detection of attacks is only increasing it becomes even more important to have more dense logs. Representing them in a tabular, ascii format is non-trivial I fully respect and understand that problem, however I would propose an exception flag of some form that if an underlying writer supports the extended attributes, they be allowed to pass, otherwise they are ignored and logged as being dropped. This, I would hope would meet everyone's needs mostly in the middle and I would assume, perhaps incorrectly, that it wouldn't be too difficult to architect in this manner. 

With all that said, I realize the priority on this is low, even for me overall at the moment. But if we're adding "nice to haves" and vision casting for future functionality in Bro, this would certainly be a helpful change that would likely be very well received.

> The facility to serialize tables to a log
> -----------------------------------------
>
>                 Key: BIT-1504
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1504
>             Project: Bro Issue Tracker
>          Issue Type: New Feature
>          Components: Bro
>    Affects Versions: 2.4
>            Reporter: Aaron Eppert
>            Priority: Low
>
> ```@load base/protocols/http/main
> @load base/protocols/http/utils
> module HTTP;
> redef record Info += {
> 	cookies: table[string] of string &optional &log;
> };
> event http_header(c: connection, is_orig: bool, name: string, value: string)
> {
> 	if ( is_orig && name == "COOKIE" ) {
> 		if ( ! c$http?$cookies ) {
> 			c$http$cookies = table();
> 		}
> 		
> 		local cookie_vec = split_string(value, /;[[:blank:]]*/);
> 		
> 		for (cookie in cookie_vec) {
> 			local kv = split_string(value, /=/);
> 			if (|kv| == 2) {
> 				c$http$cookies[kv[0]] = kv[1];
> 			}
> 		}
> 	}
> }
> ```
> Simple example. The ability to serialize the above to a log file, given it uses simple string indices and values would seem to be straight forward per looking at the Ascii and JSON writers, which appear to support TYPE_TABLE natively. I spent some time looking at how to implement this at the layers above, but the (!t->IsSet()) in SerialTypes.cc's Value::IsCompatibleType(...) is an obvious blocker and I ran out of time to deduce the rest.
> I would assume I am not alone in this want as it would make proper downstream referencing of the resulting KV pairs from the table especially easy to navigate. This is, again, very much the case when using the JSON writer given it should natively serialize into very easily usable KV pair notation.



--
This message was sent by Atlassian JIRA
(v7.0.0-OD-08-005#70107)