[Bro] Issue when adding a field to files.log

[Boreham-Smith]

Thanks Daniel,

What you suggest makes sense and explains the behaviour I observed. I guess
this leads me to the next thought - is there a way to delay the file
getting written out, or an alternate File event that could be used to
achive the outcome I am looking for?

I am happy pulling the data form the notice logs I am generating, but it
seemed tidy to have this information in the file.log too if possible.

regards,
Boreham


On Sat, Sep 12, 2015 at 3:33 PM, Daniel Thayer <dnthayer at illinois.edu>
wrote:

> What is most likely happening is that by the time your
> external program returns its result, the log record has
> already been written (without the cuckoo_id value) to files.log.
>
>
>
> On 09/11/2015 09:45 PM, Boreham-Smith wrote:
>
>> Hi All,
>>
>> I have written a script that extracts filetypes of interest, submits the
>> extracted file to the cuckoo sandbox, and records the cuckoo task_id. I
>> currently store this information successfully in the notice log, but
>> would like to add an optional field to the files.log to store this
>> task_id.
>>
>> I have confirmed that I can add and populate the new files.log field
>> with static values, but if I attempt to do this when calling an external
>> program to handle the cuckoo submission (ie I use the 'when' block
>> below), the value is not output in the log. The print statement within
>> the when block, and notice.log output confirms the value is being
>> populated, it is just not being written to files.log.
>>
>> Any suggestions on what I might be doing incorrectly?
>>
>> I have provided what I think are the relevant code extracts below, but
>> am happy to provide more detail if that will assist:
>>
>> # Add the new field to the files.log
>>
>> redef record Files::Info += {
>>      cuckoo_id: int &optional &log;
>> };
>>
>> # Function that returns the cuckoo task_id
>> function submit_cuckoo(f: fa_file): int
>> {
>>          local command = Exec::Command($cmd=fmt("%s
>> extract_files/%s",tool,f$info$extracted));
>>          return when ( local result = Exec::run(command)){
>>              local id: int  = to_int(result$stdout[0]);
>>              return id;
>>          }
>> }
>>
>> # Populate the new field
>> event file_state_remove( f: fa_file )
>> {
>>      if (f$info?$extracted) {
>>          when ( local id = submit_cuckoo(f) ){
>>              f$info$cuckoo_id = id;
>>              print fmt("Cuckoo ID value set: %d", f$info$cuckoo_id);
>>              NOTICE([$note=File::Cuckoo_Submission,
>>                      $msg=fmt("https://cuckoo/analysis/%s
>> <
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__cuckoo_analysis_-25s&d=AwMGaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Bi5qPBnY0NmYPqnRTPj_AfXQKpfQTZUpCzpfFBcawv0&m=ae24aZX-BlkOV69c9Alc2GQYRcmqia6ijosigglu5aM&s=uTipFhJT472EtdFuf9enkoihzQS0Hvht3uFGYtii2Bw&e=
>> >",
>>
>> f$info$cuckoo_id),
>>                      $f=f]);
>>          }
>>      }
>> }
>>
>>
>> # files.log extract
>> #fields ts      fuid    tx_hosts        rx_hosts        conn_uids
>> source  depth   analyzers       mime_type       filename
>> duration        local_orig      is_orig       seen_bytes
>> total_bytes     missing_bytes   overflow_bytes  timedout
>> parent_fuid     md5     sha1    sha256  extracted       cuckoo_id
>> #types  time    string  set[addr]       set[addr]       set[string]
>> string  count   set[string]     string  string  interval        bool
>> bool    count   count   countcount    bool    string  string  string
>> string  string  int
>> 1441526348.202595       FtBY2c3CsMMNsBdAil      192.168.1.xxx
>> 192.168.1.yyy   CKkqBYszNpSR6Bgaf       HTTP    0       EXTRACT
>> application/msword      -       0.108599     -F       616960  616960
>> 0       0       F       -       -       -       -
>> HTTP-FtBY2c3CsMMNsBdAil.doc     -
>>
>> # notice.log extract
>> #fields ts      uid     id.orig_h       id.orig_p       id.resp_h
>> id.resp_p       fuid    file_mime_type  file_desc       proto   note
>> msg     sub     src     dst  pn       peer_descr      actions
>> suppress_for    dropped remote_location.country_code
>> remote_location.region  remote_location.city
>> remote_location.latitude        remote_location.longitude
>> #types  time    string  addr    port    addr    port    string  string
>> string  enum    enum    string  string  addr    addr    port    count
>> string  set[enum]       interval      bool    string  string  string
>> double  double
>> 1441526362.215942       CKkqBYszNpSR6Bgaf       192.168.1.yyy   33805
>> 192.168.1.xxx   80      FtBY2c3CsMMNsBdAil      application/msword
>> http://192.168.1.xxx/files/test.doc
>> <
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__192.168.1.xxx_files_test.doc&d=AwMGaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Bi5qPBnY0NmYPqnRTPj_AfXQKpfQTZUpCzpfFBcawv0&m=ae24aZX-BlkOV69c9Alc2GQYRcmqia6ijosigglu5aM&s=C_rjh_HibNWOcOyptdaUavr_Ktn6wRtFVNCaq_cYAW4&e=
>> >
>> tcp     File::Cuckoo_Submission https://cuckoo/analysis/80
>> <
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__cuckoo_analysis_80&d=AwMGaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Bi5qPBnY0NmYPqnRTPj_AfXQKpfQTZUpCzpfFBcawv0&m=ae24aZX-BlkOV69c9Alc2GQYRcmqia6ijosigglu5aM&s=mhE5lNaIUBpgobeJZf9rZ9XlwD8p_Bjky-V2i9eheD8&e=
>> >
>> -       192.168.1.yyy   192.168.1.xxx   80      -       bro
>> Notice::ACTION_LOG      3600.000000  F-       -       -       -       -
>>
>> -------
>> regards,
>>
>> Boreham
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150912/c674fbf6/attachment.html