From seth at icir.org Thu Sep 15 07:27:20 2016 From: seth at icir.org (Seth Hall) Date: Thu, 15 Sep 2016 10:27:20 -0400 Subject: [Bro-Dev] bro-pkg -> bropkg Message-ID: <2206E1C1-6B71-427F-89B8-3FA845604030@icir.org> What does everyone think about making a change to rename bro-pkg to bropkg? I think we're early enough that we could probably get away with it and it fits with more with existing tools like broctl. I find it difficult to type the hyphen too for some reason. Thoughts? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From slagell at illinois.edu Thu Sep 15 07:35:39 2016 From: slagell at illinois.edu (Slagell, Adam J) Date: Thu, 15 Sep 2016 14:35:39 +0000 Subject: [Bro-Dev] bro-pkg -> bropkg In-Reply-To: <2206E1C1-6B71-427F-89B8-3FA845604030@icir.org> References: <2206E1C1-6B71-427F-89B8-3FA845604030@icir.org> Message-ID: <482461E8-B1EC-4A43-BE6A-28FD4F05C500@illinois.edu> Someone did come up to me and ask about that during BroCon. I am indifferent, just please let?s not open up the whole tool name naming conversation. I don?t think I can take that again. :-) > On Sep 15, 2016, at 9:27 AM, Seth Hall wrote: > > What does everyone think about making a change to rename bro-pkg to bropkg? I think we're early enough that we could probably get away with it and it fits with more with existing tools like broctl. I find it difficult to type the hyphen too for some reason. > > Thoughts? > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev ------ Adam J. Slagell Chief Information Security Officer Director, Cybersecurity Division National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." From anthony.kasza at gmail.com Thu Sep 15 09:56:16 2016 From: anthony.kasza at gmail.com (anthony kasza) Date: Thu, 15 Sep 2016 12:56:16 -0400 Subject: [Bro-Dev] Clone Failing: Sanity Check Message-ID: Is it just me or are recursive clones of the GitHub repo failing when trying to clone aux/binpac? -AK -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160915/c17595c9/attachment.html From jsiwek at illinois.edu Thu Sep 15 11:53:12 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Thu, 15 Sep 2016 18:53:12 +0000 Subject: [Bro-Dev] bro-pkg -> bropkg In-Reply-To: <482461E8-B1EC-4A43-BE6A-28FD4F05C500@illinois.edu> References: <2206E1C1-6B71-427F-89B8-3FA845604030@icir.org> <482461E8-B1EC-4A43-BE6A-28FD4F05C500@illinois.edu> Message-ID: <7E32F539-329F-4420-90B3-B52ACD062051@illinois.edu> Either way seems fine to me also. Maybe do a strawpoll and go w/ the majority. Other misc. thoughts: Internally, I already used ?bropkg? instead of ?bro-pkg? for the name of the Python package/API because Python discourages use of hyphens in module/package names. But using the Python API isn?t a common use case and figure most devs can handle doing ?import bropkg? even though the command-line tool is named ?bro-pkg?. I don?t think there?s an argument to change it for the sake of aligning the naming convention with ?broctl? because there?s already "bro-cut?. The hyphenation between those tools is already inconsistent or at least unclear why one deserves hyphenation and the other doesn?t. If easier typing is the rationale, users do already have the option of creating their own shell alias. Another idea is to keep the command line tool named ?bro-pkg?, but also install a symlink to it named ?bropkg? for convenience. - Jon > On Sep 15, 2016, at 9:35 AM, Slagell, Adam J wrote: > > Someone did come up to me and ask about that during BroCon. > > I am indifferent, just please let?s not open up the whole tool name naming conversation. I don?t think I can take that again. :-) > >> On Sep 15, 2016, at 9:27 AM, Seth Hall wrote: >> >> What does everyone think about making a change to rename bro-pkg to bropkg? I think we're early enough that we could probably get away with it and it fits with more with existing tools like broctl. I find it difficult to type the hyphen too for some reason. >> >> Thoughts? >> >> .Seth >> >> -- >> Seth Hall >> International Computer Science Institute >> (Bro) because everyone has a network >> http://www.bro.org/ >> >> >> _______________________________________________ >> bro-dev mailing list >> bro-dev at bro.org >> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > > ------ > > Adam J. Slagell > Chief Information Security Officer > Director, Cybersecurity Division > National Center for Supercomputing Applications > University of Illinois at Urbana-Champaign > www.slagell.info > > "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." > > > > > > > > > > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From michalpurzynski1 at gmail.com Thu Sep 15 17:25:12 2016 From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=) Date: Thu, 15 Sep 2016 19:25:12 -0500 Subject: [Bro-Dev] bro-pkg -> bropkg In-Reply-To: <7E32F539-329F-4420-90B3-B52ACD062051@illinois.edu> References: <2206E1C1-6B71-427F-89B8-3FA845604030@icir.org> <482461E8-B1EC-4A43-BE6A-28FD4F05C500@illinois.edu> <7E32F539-329F-4420-90B3-B52ACD062051@illinois.edu> Message-ID: Bropkg is easier to type indeed. And then let's change the brocut. > On 15 Sep 2016, at 13:53, Siwek, Jon wrote: > > Either way seems fine to me also. Maybe do a strawpoll and go w/ the majority. Other misc. thoughts: > > Internally, I already used ?bropkg? instead of ?bro-pkg? for the name of the Python package/API because Python discourages use of hyphens in module/package names. But using the Python API isn?t a common use case and figure most devs can handle doing ?import bropkg? even though the command-line tool is named ?bro-pkg?. > > I don?t think there?s an argument to change it for the sake of aligning the naming convention with ?broctl? because there?s already "bro-cut?. The hyphenation between those tools is already inconsistent or at least unclear why one deserves hyphenation and the other doesn?t. > > If easier typing is the rationale, users do already have the option of creating their own shell alias. > > Another idea is to keep the command line tool named ?bro-pkg?, but also install a symlink to it named ?bropkg? for convenience. > > - Jon > >> On Sep 15, 2016, at 9:35 AM, Slagell, Adam J wrote: >> >> Someone did come up to me and ask about that during BroCon. >> >> I am indifferent, just please let?s not open up the whole tool name naming conversation. I don?t think I can take that again. :-) >> >>> On Sep 15, 2016, at 9:27 AM, Seth Hall wrote: >>> >>> What does everyone think about making a change to rename bro-pkg to bropkg? I think we're early enough that we could probably get away with it and it fits with more with existing tools like broctl. I find it difficult to type the hyphen too for some reason. >>> >>> Thoughts? >>> >>> .Seth >>> >>> -- >>> Seth Hall >>> International Computer Science Institute >>> (Bro) because everyone has a network >>> http://www.bro.org/ >>> >>> >>> _______________________________________________ >>> bro-dev mailing list >>> bro-dev at bro.org >>> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev >> >> ------ >> >> Adam J. Slagell >> Chief Information Security Officer >> Director, Cybersecurity Division >> National Center for Supercomputing Applications >> University of Illinois at Urbana-Champaign >> www.slagell.info >> >> "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> bro-dev mailing list >> bro-dev at bro.org >> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > > > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160915/abf7edfa/attachment.html From robin at icir.org Fri Sep 16 13:44:04 2016 From: robin at icir.org (Robin Sommer) Date: Fri, 16 Sep 2016 13:44:04 -0700 Subject: [Bro-Dev] Clone Failing: Sanity Check In-Reply-To: References: Message-ID: <20160916204404.GD5321@icir.org> On Thu, Sep 15, 2016 at 12:56 -0400, you wrote: > Is it just me or are recursive clones of the GitHub repo failing when > trying to clone aux/binpac? Just tried and it worked fine for me. If you still see the problem, what's the error you're getting? Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From robin at icir.org Fri Sep 16 13:57:38 2016 From: robin at icir.org (Robin Sommer) Date: Fri, 16 Sep 2016 13:57:38 -0700 Subject: [Bro-Dev] bro-pkg -> bropkg In-Reply-To: References: <2206E1C1-6B71-427F-89B8-3FA845604030@icir.org> <482461E8-B1EC-4A43-BE6A-28FD4F05C500@illinois.edu> <7E32F539-329F-4420-90B3-B52ACD062051@illinois.edu> Message-ID: <20160916205738.GI5321@icir.org> On Thu, Sep 15, 2016 at 19:25 -0500, you wrote: > Bropkg is easier to type indeed. And then let's change the brocut. No strong preference either way on my end, but let's not touch bro-cut please, that's used in too many scripts. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From vallentin at icir.org Fri Sep 16 14:17:12 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Fri, 16 Sep 2016 14:17:12 -0700 Subject: [Bro-Dev] bro-pkg -> bropkg In-Reply-To: References: <2206E1C1-6B71-427F-89B8-3FA845604030@icir.org> <482461E8-B1EC-4A43-BE6A-28FD4F05C500@illinois.edu> <7E32F539-329F-4420-90B3-B52ACD062051@illinois.edu> Message-ID: <20160916211712.GE40210@samurai.ICIR.org> > Bropkg is easier to type indeed. And then let's change the brocut. Yeah, all bro* utils should be consistent. If we change bro-pkg, then bro-cut must undergo the same changes. Personally, I never had trouble typing "bro-pkg" though. Can we do a Twitter poll to figure out what our users prefer? Matthias From jan.grashoefer at gmail.com Fri Sep 16 14:21:20 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Fri, 16 Sep 2016 23:21:20 +0200 Subject: [Bro-Dev] bro-pkg -> bropkg In-Reply-To: <20160916205738.GI5321@icir.org> References: <2206E1C1-6B71-427F-89B8-3FA845604030@icir.org> <482461E8-B1EC-4A43-BE6A-28FD4F05C500@illinois.edu> <7E32F539-329F-4420-90B3-B52ACD062051@illinois.edu> <20160916205738.GI5321@icir.org> Message-ID: <2a9f6a6a-a798-1f5c-fcda-01b197cce6b3@gmail.com> > No strong preference either way on my end, but let's not touch bro-cut > please, that's used in too many scripts. I like Jon's idea to create a symlink for bro-pkg. One could also install links for brocut and bro-ctl. From vallentin at icir.org Fri Sep 16 14:54:15 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Fri, 16 Sep 2016 14:54:15 -0700 Subject: [Bro-Dev] bro-pkg -> bropkg In-Reply-To: <2a9f6a6a-a798-1f5c-fcda-01b197cce6b3@gmail.com> References: <2206E1C1-6B71-427F-89B8-3FA845604030@icir.org> <482461E8-B1EC-4A43-BE6A-28FD4F05C500@illinois.edu> <7E32F539-329F-4420-90B3-B52ACD062051@illinois.edu> <20160916205738.GI5321@icir.org> <2a9f6a6a-a798-1f5c-fcda-01b197cce6b3@gmail.com> Message-ID: <20160916215415.GF40210@samurai.ICIR.org> > I like Jon's idea to create a symlink for bro-pkg. One could also > install links for brocut and bro-ctl. PREFIX/bin/bro PREFIX/bin/bro-ctl PREFIX/bin/broctl PREFIX/bin/bro-cut PREFIX/bin/brocut PREFIX/bin/bro-pkg PREFIX/bin/bropkg .. That looks too confusing to me! My vote is to go with one scheme, ideally the one that breaks less existing tooling. Matthias From anthony.kasza at gmail.com Fri Sep 16 15:10:18 2016 From: anthony.kasza at gmail.com (anthony kasza) Date: Fri, 16 Sep 2016 18:10:18 -0400 Subject: [Bro-Dev] Clone Failing: Sanity Check In-Reply-To: <20160916204404.GD5321@icir.org> References: <20160916204404.GD5321@icir.org> Message-ID: It's working now, yes. -AK On Sep 16, 2016 4:44 PM, "Robin Sommer" wrote: > > > On Thu, Sep 15, 2016 at 12:56 -0400, you wrote: > > > Is it just me or are recursive clones of the GitHub repo failing when > > trying to clone aux/binpac? > > Just tried and it worked fine for me. If you still see the problem, > what's the error you're getting? > > Robin > > -- > Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160916/1078ecc3/attachment.html From slagell at illinois.edu Fri Sep 16 15:35:02 2016 From: slagell at illinois.edu (Slagell, Adam J) Date: Fri, 16 Sep 2016 22:35:02 +0000 Subject: [Bro-Dev] bro-pkg -> bropkg In-Reply-To: <20160916211712.GE40210@samurai.ICIR.org> References: <2206E1C1-6B71-427F-89B8-3FA845604030@icir.org> <482461E8-B1EC-4A43-BE6A-28FD4F05C500@illinois.edu> <7E32F539-329F-4420-90B3-B52ACD062051@illinois.edu> , <20160916211712.GE40210@samurai.ICIR.org> Message-ID: Let's hold off if we might be renaming Bro On Sep 16, 2016, at 4:17 PM, Matthias Vallentin wrote: >> Bropkg is easier to type indeed. And then let's change the brocut. > > Yeah, all bro* utils should be consistent. If we change bro-pkg, then > bro-cut must undergo the same changes. > > Personally, I never had trouble typing "bro-pkg" though. Can we do a > Twitter poll to figure out what our users prefer? > > Matthias > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From robin at icir.org Mon Sep 19 16:27:54 2016 From: robin at icir.org (Robin Sommer) Date: Mon, 19 Sep 2016 16:27:54 -0700 Subject: [Bro-Dev] Proposal: Operating the Bro package manager offline Message-ID: <20160919232754.GD94635@icir.org> At BroCon several people pointed out a need to install Bro packages on machines that do not have a direct external connection. One idea would be some kind of proxy scheme where an intermediary git repository mirrors packages locally; bro-pkg would then pull from there. However, I don't think any of us really liked that idea much. After a few rounds of discussions, Seth and I came up with a different idea that seems easier to manage: extending bro-pkg to bundle packages into deployment files that can be easily pushed to Bro systems simply by copying them over. I?ve tried to flesh this out a bit more, and would be interested to hear what you all think about this approach. And @Jon: Do you think this would be doable that way? Here?s the idea: 1. Generally, one first uses bro-pkg as usual to install packages onto a local Bro system that does have external Internet connectivity (this could be just a dummy Bro installation). One installs new packages there, updates existing ones, etc., until reaching a state that one wants to push out to the actual Bro system. 2. We add a new ?bundle? command to bro-pkg that serializes the current state of packages into a single file on disk, a ?package bundle?. The bundle contains the complete content of all currently installed packages, using some kind of suitable container format (could be just a ZIP file, or whatever works; the internal representation doesn?t really matter). 3. Users create such a bundle on the local system and then simply copy that bundle file over to all target Bro machines that do not have a external connectivity themselves, using whatever mechanism they have available (e.g., just scp; or maybe through some configuration management system like Ansible etc.). 4. On the target machine, one runs a corresponding ?bro-pkg unbundle? command on that bundle file. That command will completely replace the system?s current set of packages with the bundle?s content. As a result, that machine will now have exactly the same packages installed as the original system. This would be the general scheme. A couple of people I talked to at BroCon confirmed that this would offer a viable solution for them, and that they would indeed much prefer copying files around over maintaining local git mirrors. Some additional thoughts on variations/extensions of this basic scheme: - I?m not quite sure if the bundle should contain just the packages themselves or further bro-pkg state as well, such as which packages are currently loaded. Right now I?m learning towards saying ?just the packages?; that would basically treat bundles just as a transport mechanism to get packages over to another box. The actual Bro machines would still keep control over which packages to actually load, etc. - As it is described above, Step 1 would require having a local Bro installation into which packages get installed before they can be bundled up. It would be nice to have a mode where bro-pkg can operate without having a Bro around at all, just downloading packages locally somewhere for bundling them up. I could also see offering an even simpler mode where one simply lists packages to bundle on the command-line: ?bro-pkg bundle ?. That would be particularly useful with configuration management systems I think. - It would be neat if bro-pkg's Python library exposed operations to inspect & retrieve the content of a bundle, such as iterating over the packages inside a bundle and iterating over the files inside a package. That way one could easily build target-side scripts that process and validate bundles before going ahead and installing them (e.g., imposing custom restrictions on what kind of packages one allows to put in place; or ensuring that a bundle always contains a set of packages the site deems mandatory, to avoid configuration mistakes; or even just logging what gets pushed out). What do you guys think about this? Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From michalpurzynski1 at gmail.com Mon Sep 19 20:40:29 2016 From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=) Date: Mon, 19 Sep 2016 22:40:29 -0500 Subject: [Bro-Dev] Proposal: Operating the Bro package manager offline In-Reply-To: <20160919232754.GD94635@icir.org> References: <20160919232754.GD94635@icir.org> Message-ID: <7491C940-234F-4359-B4EF-D76029B4FEA0@gmail.com> One can, depending on the level of paranoia or risk management: Use a proxy for accessing git repos with packages. For GitHub it presents a problem - because github is essential an arbitrary file upload service and without terminating SSL one cannot create good enough proxy rules around it. What we do is we create a mirror of GitHub repositories that copy only white listed parts. This is beyond the package manager project. I really like the idea of being able to bundle - cap - unbundle package and I think it should be supported. With possibly a configuration file to tell "bundle these packages" only. > On 19 Sep 2016, at 18:27, Robin Sommer wrote: > > At BroCon several people pointed out a need to install Bro packages on > machines that do not have a direct external connection. One idea would > be some kind of proxy scheme where an intermediary git repository > mirrors packages locally; bro-pkg would then pull from there. However, > I don't think any of us really liked that idea much. After a few > rounds of discussions, Seth and I came up with a different idea that > seems easier to manage: extending bro-pkg to bundle packages into > deployment files that can be easily pushed to Bro systems simply by > copying them over. > > I?ve tried to flesh this out a bit more, and would be interested to > hear what you all think about this approach. And @Jon: Do you think > this would be doable that way? > > Here?s the idea: > > 1. Generally, one first uses bro-pkg as usual to install packages onto > a local Bro system that does have external Internet connectivity > (this could be just a dummy Bro installation). One installs new > packages there, updates existing ones, etc., until reaching a state > that one wants to push out to the actual Bro system. > > 2. We add a new ?bundle? command to bro-pkg that serializes the > current state of packages into a single file on disk, a ?package > bundle?. The bundle contains the complete content of all currently > installed packages, using some kind of suitable container format > (could be just a ZIP file, or whatever works; the internal > representation doesn?t really matter). > > 3. Users create such a bundle on the local system and then simply copy > that bundle file over to all target Bro machines that do not have a > external connectivity themselves, using whatever mechanism they > have available (e.g., just scp; or maybe through some configuration > management system like Ansible etc.). > > 4. On the target machine, one runs a corresponding ?bro-pkg unbundle? > command on that bundle file. That command will completely replace > the system?s current set of packages with the bundle?s content. As > a result, that machine will now have exactly the same packages > installed as the original system. > > This would be the general scheme. A couple of people I talked to at > BroCon confirmed that this would offer a viable solution for them, and > that they would indeed much prefer copying files around over > maintaining local git mirrors. > > Some additional thoughts on variations/extensions of this basic scheme: > > - I?m not quite sure if the bundle should contain just the packages > themselves or further bro-pkg state as well, such as which packages > are currently loaded. Right now I?m learning towards saying ?just > the packages?; that would basically treat bundles just as a > transport mechanism to get packages over to another box. The actual > Bro machines would still keep control over which packages to > actually load, etc. > > - As it is described above, Step 1 would require having a local Bro > installation into which packages get installed before they can be > bundled up. It would be nice to have a mode where bro-pkg can > operate without having a Bro around at all, just downloading > packages locally somewhere for bundling them up. I could also see > offering an even simpler mode where one simply lists packages to > bundle on the command-line: ?bro-pkg bundle ?. > That would be particularly useful with configuration management > systems I think. > > - It would be neat if bro-pkg's Python library exposed operations to > inspect & retrieve the content of a bundle, such as iterating over > the packages inside a bundle and iterating over the files inside a > package. That way one could easily build target-side scripts that > process and validate bundles before going ahead and installing them > (e.g., imposing custom restrictions on what kind of packages one > allows to put in place; or ensuring that a bundle always contains a > set of packages the site deems mandatory, to avoid configuration > mistakes; or even just logging what gets pushed out). > > What do you guys think about this? > > Robin > > -- > Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160919/dd9651ec/attachment.html From jazoff at illinois.edu Mon Sep 19 21:11:39 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 20 Sep 2016 04:11:39 +0000 Subject: [Bro-Dev] Proposal: Operating the Bro package manager offline In-Reply-To: <20160919232754.GD94635@icir.org> References: <20160919232754.GD94635@icir.org> Message-ID: <84732D08-4E1E-4D95-AF93-207EB926FA49@illinois.edu> Sounds great.. What you are describing is basically source and binary packages. The only thing I would do differently is that you wouldn't want bundles (at least not as the only feature) but individual source and binary packages. For example bro-pkg install sethhall/domain-tld would still install the package from git like normal, but one could do bro-pkg dist sethhall/domain-tld or just bro-pkg dist # inside the project and this would generate a versioned sethhall_domain-tld-1.0.0.tar.gz source package Then, that file could be copied over to the target machine and bro-pkg install sethhall_domain-tld-1.0.0.tar.gz or variations of bro-pkg install https://packages.internal/bro/sethhall_domain-tld-1.0.0.tar.gz would install it. The hard part is how to handle compiled architecture specific packages. For something like a myricom plugin, bro-pkg dist bro-plugins/myricom would generate a bro-plugins_myricom-1.0.0.tar.gz source package as before If copied over to the target system bro-pkg install bro-plugins_myricom-1.0.0.tar.gz would build and install it as long as build tools were installed. If 'no build tools on servers' is a requirement, then bro-pkg build bro-plugins/myricom on a build server would compile and generate something like bro-plugins_myricom-1.0.0-bro2.5-linux-amd64.tar.gz And with some more glue, could easily build .deb and .rpm packages. This is basically how python/pip/wheel works now (after ~10 years of being pretty bad). You could still have a bundle command as a very small layer on top of packages, but the building block should be an individual package distribution. Also, > That command will completely replace the system?s current set of packages with the bundle?s content should probably not be the only behavior. Installing should be the default unless you do something like bro-pkg install --replace my.bro.bundle otherwise it would be impossible to compose bundles. -- - Justin Azoff > On Sep 19, 2016, at 7:27 PM, Robin Sommer wrote: > > At BroCon several people pointed out a need to install Bro packages on > machines that do not have a direct external connection. One idea would > be some kind of proxy scheme where an intermediary git repository > mirrors packages locally; bro-pkg would then pull from there. However, > I don't think any of us really liked that idea much. After a few > rounds of discussions, Seth and I came up with a different idea that > seems easier to manage: extending bro-pkg to bundle packages into > deployment files that can be easily pushed to Bro systems simply by > copying them over. > > I?ve tried to flesh this out a bit more, and would be interested to > hear what you all think about this approach. And @Jon: Do you think > this would be doable that way? > > Here?s the idea: > > 1. Generally, one first uses bro-pkg as usual to install packages onto > a local Bro system that does have external Internet connectivity > (this could be just a dummy Bro installation). One installs new > packages there, updates existing ones, etc., until reaching a state > that one wants to push out to the actual Bro system. > > 2. We add a new ?bundle? command to bro-pkg that serializes the > current state of packages into a single file on disk, a ?package > bundle?. The bundle contains the complete content of all currently > installed packages, using some kind of suitable container format > (could be just a ZIP file, or whatever works; the internal > representation doesn?t really matter). > > 3. Users create such a bundle on the local system and then simply copy > that bundle file over to all target Bro machines that do not have a > external connectivity themselves, using whatever mechanism they > have available (e.g., just scp; or maybe through some configuration > management system like Ansible etc.). > > 4. On the target machine, one runs a corresponding ?bro-pkg unbundle? > command on that bundle file. That command will completely replace > the system?s current set of packages with the bundle?s content. As > a result, that machine will now have exactly the same packages > installed as the original system. > > This would be the general scheme. A couple of people I talked to at > BroCon confirmed that this would offer a viable solution for them, and > that they would indeed much prefer copying files around over > maintaining local git mirrors. > > Some additional thoughts on variations/extensions of this basic scheme: > > - I?m not quite sure if the bundle should contain just the packages > themselves or further bro-pkg state as well, such as which packages > are currently loaded. Right now I?m learning towards saying ?just > the packages?; that would basically treat bundles just as a > transport mechanism to get packages over to another box. The actual > Bro machines would still keep control over which packages to > actually load, etc. > > - As it is described above, Step 1 would require having a local Bro > installation into which packages get installed before they can be > bundled up. It would be nice to have a mode where bro-pkg can > operate without having a Bro around at all, just downloading > packages locally somewhere for bundling them up. I could also see > offering an even simpler mode where one simply lists packages to > bundle on the command-line: ?bro-pkg bundle ?. > That would be particularly useful with configuration management > systems I think. > > - It would be neat if bro-pkg's Python library exposed operations to > inspect & retrieve the content of a bundle, such as iterating over > the packages inside a bundle and iterating over the files inside a > package. That way one could easily build target-side scripts that > process and validate bundles before going ahead and installing them > (e.g., imposing custom restrictions on what kind of packages one > allows to put in place; or ensuring that a bundle always contains a > set of packages the site deems mandatory, to avoid configuration > mistakes; or even just logging what gets pushed out). > > What do you guys think about this? > > Robin From jsiwek at illinois.edu Tue Sep 20 07:35:29 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Tue, 20 Sep 2016 14:35:29 +0000 Subject: [Bro-Dev] Proposal: Operating the Bro package manager offline In-Reply-To: <20160919232754.GD94635@icir.org> References: <20160919232754.GD94635@icir.org> Message-ID: <53ACD779-5BA5-4F06-AC88-0A7FC76D69D5@illinois.edu> > On Sep 19, 2016, at 6:27 PM, Robin Sommer wrote: > > @Jon: Do you think this would be doable that way? Yeah, looks viable. > - I?m not quite sure if the bundle should contain just the packages > themselves or further bro-pkg state as well, such as which packages > are currently loaded. Right now I?m learning towards saying ?just > the packages?; that would basically treat bundles just as a > transport mechanism to get packages over to another box. The actual > Bro machines would still keep control over which packages to > actually load, etc. Yeah, I think that?s generally how it would be also. Though, maybe since the default behavior of the ?install? command is to automatically do a subsequent ?load? it makes sense to automatically do loads after ?unbundle? as well. It would then be up to user whether they actually ?@load packages? in the target machines local.bro or just pick and choose, so they still have complete control over loading/unloading. > - As it is described above, Step 1 would require having a local Bro > installation into which packages get installed before they can be > bundled up. It would be nice to have a mode where bro-pkg can > operate without having a Bro around at all, just downloading > packages locally somewhere for bundling them up. I could also see > offering an even simpler mode where one simply lists packages to > bundle on the command-line: ?bro-pkg bundle ?. > That would be particularly useful with configuration management > systems I think. I think bro-pkg currently works fine even if you don?t have a local Bro installation? If you build plugins, you?d need Bro source code, but don?t actually need it installed. Then on the target machine, ?unbundle? just unpacks into whatever bro-pkg paths are configured for script_dir/plugin_dir. - Jon From jsiwek at illinois.edu Tue Sep 20 07:46:35 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Tue, 20 Sep 2016 14:46:35 +0000 Subject: [Bro-Dev] Proposal: Operating the Bro package manager offline In-Reply-To: <84732D08-4E1E-4D95-AF93-207EB926FA49@illinois.edu> References: <20160919232754.GD94635@icir.org> <84732D08-4E1E-4D95-AF93-207EB926FA49@illinois.edu> Message-ID: <8BEE9B8D-6CBD-4635-B9B4-AFDF10B284A4@illinois.edu> > On Sep 19, 2016, at 11:11 PM, Azoff, Justin S wrote: > > Sounds great.. What you are describing is basically source and binary packages. In concept, I like the idea of simply extending the ?install? command to be able to install from a source or binary packages, but how would the package update process look like for users? E.g. with the bundle/unbundle the update process would be: # On source machine $ bro-pkg refresh $ bro-pkg update ?all $ bro-pkg bundle # Copy bundle to target machine # On target machine $ bro-pkg unbundle With that approach, the user never has to consider individual packages so updating is still straight-forward. But with the approach of being able to install specific versions of packages from a source/binary distribution, how do you make it simple for a user to update to newer versions when absent an internet connection? - Jon From robin at icir.org Tue Sep 20 12:27:18 2016 From: robin at icir.org (Robin Sommer) Date: Tue, 20 Sep 2016 12:27:18 -0700 Subject: [Bro-Dev] Proposal: Operating the Bro package manager offline In-Reply-To: <84732D08-4E1E-4D95-AF93-207EB926FA49@illinois.edu> References: <20160919232754.GD94635@icir.org> <84732D08-4E1E-4D95-AF93-207EB926FA49@illinois.edu> Message-ID: <20160920192718.GD32496@icir.org> On Tue, Sep 20, 2016 at 04:11 +0000, you wrote: > Sounds great.. What you are describing is basically source and binary > packages. Not sure "binary package" is the right term, at least for the first part of your description (where nothing gets built). The difference seems more to be having one thing (the bundle) containing all of the packages, vs. creating individual package files to distribute. I would like the bundle approach more because it means one can more easily recreate the same state on multiple servers; all the package management happens locally on one system and the final state just gets pushed out. That's in contrast to manually ensuring that all the target systems now end up with the right set of packages. For packages that need to be built I can see that real binary packages would be useful too (like indeed in the "no build tools on server" setting), but that sounds like an orthogonal feature (and more complex to add to the package manager). Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From robin at icir.org Tue Sep 20 12:32:37 2016 From: robin at icir.org (Robin Sommer) Date: Tue, 20 Sep 2016 12:32:37 -0700 Subject: [Bro-Dev] Proposal: Operating the Bro package manager offline In-Reply-To: <53ACD779-5BA5-4F06-AC88-0A7FC76D69D5@illinois.edu> References: <20160919232754.GD94635@icir.org> <53ACD779-5BA5-4F06-AC88-0A7FC76D69D5@illinois.edu> Message-ID: <20160920193237.GE32496@icir.org> On Tue, Sep 20, 2016 at 14:35 +0000, you wrote: > Yeah, I think that?s generally how it would be also. Though, maybe > since the default behavior of the ?install? command is to > automatically do a subsequent ?load? it makes sense to automatically > do loads after ?unbundle? as well. What would that do for updates? Say I've unloaded a package, but I'm still updating it? That shouldn't enable it, so would it do that implicit "load" only on first install? > I think bro-pkg currently works fine even if you don?t have a local Bro installation? I was thinking that it needs bro-config. Is that only for "autoconfig" to set up the right paths? If so, then maybe we can add an option to "autoconfig" to setup (and create) local paths for this working-without-a-Bro mode? Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jsiwek at illinois.edu Wed Sep 21 07:47:09 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Wed, 21 Sep 2016 14:47:09 +0000 Subject: [Bro-Dev] Proposal: Operating the Bro package manager offline In-Reply-To: <20160920193237.GE32496@icir.org> References: <20160919232754.GD94635@icir.org> <53ACD779-5BA5-4F06-AC88-0A7FC76D69D5@illinois.edu> <20160920193237.GE32496@icir.org> Message-ID: <627BEB79-B296-4DFD-9153-52C049BA8779@illinois.edu> > On Sep 20, 2016, at 2:32 PM, Robin Sommer wrote: > > On Tue, Sep 20, 2016 at 14:35 +0000, you wrote: > >> Yeah, I think that?s generally how it would be also. Though, maybe >> since the default behavior of the ?install? command is to >> automatically do a subsequent ?load? it makes sense to automatically >> do loads after ?unbundle? as well. > > What would that do for updates? Say I've unloaded a package, but I'm > still updating it? That shouldn't enable it, so would it do that > implicit "load" only on first install? Yeah, I imagine it detecting whether a given package in a bundle is already installed. If it is, then just install/overwrite it without changing its ?loaded? status. If not, then install and also load. >> I think bro-pkg currently works fine even if you don?t have a local Bro installation? > > I was thinking that it needs bro-config. Is that only for "autoconfig" > to set up the right paths? Yes, bro-config is only necessary for the ?autoconfig? command. > If so, then maybe we can add an option to > "autoconfig" to setup (and create) local paths for this > working-without-a-Bro mode? Currently for that use-case, I?d say ?just don?t run autoconfig, the default paths are sufficient?. On the source machine, packages can get installed anywhere. On the destination machine, the unbundling process will relocate everything to the correct paths. In other words, the installation paths of a bundle are not hardcoded within it. But if we want to keep the bro-pkg initial setup procedure more consistent, I can have autoconfig first ask a question: ?Will you be using this system to create package bundles?? If no, proceed with the current autoconfig logic (use bro-config to setup paths). If yes, still proceed with autoconfig if bro-config is available, else no-op. - Jon From robin at icir.org Wed Sep 21 07:54:52 2016 From: robin at icir.org (Robin Sommer) Date: Wed, 21 Sep 2016 07:54:52 -0700 Subject: [Bro-Dev] Proposal: Operating the Bro package manager offline In-Reply-To: <627BEB79-B296-4DFD-9153-52C049BA8779@illinois.edu> References: <20160919232754.GD94635@icir.org> <53ACD779-5BA5-4F06-AC88-0A7FC76D69D5@illinois.edu> <20160920193237.GE32496@icir.org> <627BEB79-B296-4DFD-9153-52C049BA8779@illinois.edu> Message-ID: <20160921145452.GO32496@icir.org> On Wed, Sep 21, 2016 at 14:47 +0000, you wrote: > Yeah, I imagine it detecting whether a given package in a bundle is > already installed. If it is, then just install/overwrite it without > changing its ?loaded? status. If not, then install and also load. Yeah, makes sense. > Currently for that use-case, I?d say ?just don?t run autoconfig, the > default paths are sufficient?. What are the default paths? In other words, where do the downloaded packages get put if I don't set anything further up? Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From VARD.ANTINYAN at cse.gu.se Wed Sep 21 08:17:31 2016 From: VARD.ANTINYAN at cse.gu.se (Vard Antinyan) Date: Wed, 21 Sep 2016 15:17:31 +0000 Subject: [Bro-Dev] Code complexity survey Message-ID: <024e756408be4bcb98d2a1d4175f2056@cse.gu.se> Dear developers, We have undertaken a task to assess code complexity triggers and generate recommendations for developing simple and understandable code. Our intension is to share the results with you, developers, so everyone can learn the triggers behind complex software. We need your help for rigorous results. My request to you is - if you get 5-10 min. time, would you please consider to answer the questions of this survey on code complexity? https://goo.gl/forms/h9WXZ8VSEw7BUyHg1 You are welcome to learn preliminary results through this link: https://www.facebook.com/SoftwareCodeQuality/photos/?tab=album&album_id=1639816749664288 The results will be shared in a public webpage and everyone possible will be invited to learn and discuss them. Your knowledge and experience is vital for achieving substantial and generalizable results, and your effort is much appreciated! Sincerely Vard Antinyan PhD candidate in University of Gothenburg, Sweden Tel: 0046317725707 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160921/37641457/attachment.html From jsiwek at illinois.edu Wed Sep 21 10:51:52 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Wed, 21 Sep 2016 17:51:52 +0000 Subject: [Bro-Dev] Proposal: Operating the Bro package manager offline In-Reply-To: <20160921145452.GO32496@icir.org> References: <20160919232754.GD94635@icir.org> <53ACD779-5BA5-4F06-AC88-0A7FC76D69D5@illinois.edu> <20160920193237.GE32496@icir.org> <627BEB79-B296-4DFD-9153-52C049BA8779@illinois.edu> <20160921145452.GO32496@icir.org> Message-ID: <0FFE4371-975A-4020-AA81-F32AAD18D85B@illinois.edu> > On Sep 21, 2016, at 9:54 AM, Robin Sommer wrote: > > What are the default paths? In other words, where do the downloaded > packages get put if I don't set anything further up? They get put within ~/.bro-pkg/ (The Advanced Configuration [1] section has more usage info related to that). - Jon [1] http://bro-package-manager.readthedocs.io/en/stable/quickstart.html#advanced-configuration From robin at icir.org Wed Sep 21 12:45:46 2016 From: robin at icir.org (Robin Sommer) Date: Wed, 21 Sep 2016 12:45:46 -0700 Subject: [Bro-Dev] Proposal: Operating the Bro package manager offline In-Reply-To: <20160919232754.GD94635@icir.org> References: <20160919232754.GD94635@icir.org> Message-ID: <20160921194546.GD37771@icir.org> After discussion with some more folks, and the email thread, some further tweaks to my original "bundle" proposal: - Following Justin's suggestion, unbundling should by default not replace everything currently installed; and instead offer a "--replace" option if one wants that. - People seem to like the "bro-pkg bundle " approach quite a bit as well, so that should be a "first class citizen" too. So we'd have two modes: either (1) bundle what's currently installed locally, or (2) bundle what's given explicitly at startup (and ignore the current local state completely). For the latter mode, allowing the list of packages to come from a configuration file (rather than on the command line) would be very useful as well I hear. - Per Jon's suggestion, when unbundling a first-time install of a package autoloads it. - I'm sure this would just work automatically, but for completeness: unbundling should install packages just exactly the same way as if they had been coming in "normally" from git. In particular, if that target system later does get external connectivity and switches to using git directly, it should "just work". - Also for completness: The "bundle mode" should work without requiring a local Bro installation. Per Jon's reply, that would be the case automatically already as well. - One clarification, as I'm not sure that was clear: "bundle" would not build any architecture-specific code locally, that would happen at unbundle time on the target server. "bundle" is really just a transport mechanism for the getting package source content over to offline systems. - I'm withdrawing the part on exposing an introspection API for bundles through Python. I wouldn't object to having that, but it's probably unnecessary given that one can also just look at the package directory once things are installed. Should be good enough. Robin On Mon, Sep 19, 2016 at 16:27 -0700, I wrote: > At BroCon several people pointed out a need to install Bro packages on > machines that do not have a direct external connection. One idea would > be some kind of proxy scheme where an intermediary git repository > mirrors packages locally; bro-pkg would then pull from there. However, > I don't think any of us really liked that idea much. After a few > rounds of discussions, Seth and I came up with a different idea that > seems easier to manage: extending bro-pkg to bundle packages into > deployment files that can be easily pushed to Bro systems simply by > copying them over. > > I?ve tried to flesh this out a bit more, and would be interested to > hear what you all think about this approach. And @Jon: Do you think > this would be doable that way? > > Here?s the idea: > > 1. Generally, one first uses bro-pkg as usual to install packages onto > a local Bro system that does have external Internet connectivity > (this could be just a dummy Bro installation). One installs new > packages there, updates existing ones, etc., until reaching a state > that one wants to push out to the actual Bro system. > > 2. We add a new ?bundle? command to bro-pkg that serializes the > current state of packages into a single file on disk, a ?package > bundle?. The bundle contains the complete content of all currently > installed packages, using some kind of suitable container format > (could be just a ZIP file, or whatever works; the internal > representation doesn?t really matter). > > 3. Users create such a bundle on the local system and then simply copy > that bundle file over to all target Bro machines that do not have a > external connectivity themselves, using whatever mechanism they > have available (e.g., just scp; or maybe through some configuration > management system like Ansible etc.). > > 4. On the target machine, one runs a corresponding ?bro-pkg unbundle? > command on that bundle file. That command will completely replace > the system?s current set of packages with the bundle?s content. As > a result, that machine will now have exactly the same packages > installed as the original system. > > This would be the general scheme. A couple of people I talked to at > BroCon confirmed that this would offer a viable solution for them, and > that they would indeed much prefer copying files around over > maintaining local git mirrors. > > Some additional thoughts on variations/extensions of this basic scheme: > > - I?m not quite sure if the bundle should contain just the packages > themselves or further bro-pkg state as well, such as which packages > are currently loaded. Right now I?m learning towards saying ?just > the packages?; that would basically treat bundles just as a > transport mechanism to get packages over to another box. The actual > Bro machines would still keep control over which packages to > actually load, etc. > > - As it is described above, Step 1 would require having a local Bro > installation into which packages get installed before they can be > bundled up. It would be nice to have a mode where bro-pkg can > operate without having a Bro around at all, just downloading > packages locally somewhere for bundling them up. I could also see > offering an even simpler mode where one simply lists packages to > bundle on the command-line: ?bro-pkg bundle ?. > That would be particularly useful with configuration management > systems I think. > > - It would be neat if bro-pkg's Python library exposed operations to > inspect & retrieve the content of a bundle, such as iterating over > the packages inside a bundle and iterating over the files inside a > package. That way one could easily build target-side scripts that > process and validate bundles before going ahead and installing them > (e.g., imposing custom restrictions on what kind of packages one > allows to put in place; or ensuring that a bundle always contains a > set of packages the site deems mandatory, to avoid configuration > mistakes; or even just logging what gets pushed out). > > What do you guys think about this? > > Robin > > -- > Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From mfernandez at mitre.org Wed Sep 21 14:03:27 2016 From: mfernandez at mitre.org (Fernandez, Mark I) Date: Wed, 21 Sep 2016 21:03:27 +0000 Subject: [Bro-Dev] ICAP Analyzer Design Guidance Message-ID: I am reviewing my source code and scripts for the ICAP Analyzer that I presented last week at BroCon, with the intent of releasing the new analyzer to the Bro community. There is one key aspect which I designed a certain way, but I wonder if it would be acceptable by the community or if it would introduce problems. I appreciate your feedback. In the 'main.bro' script for the ICAP Analyzer, I redefine the 'conn_id' record to include a new element, as follows: redef record conn_id += { orig_u : string &log &optional; } where 'orig_u' is derived from the ICAP header 'X-Authenticated-User' and is associated with the userid on the local domain that originated the HTTP request. At the time I wrote the code, it made perfect sense to extend the 'conn_id' record to include the 'orig_u' element, and it works very well in my operational environment. However, now that I am preparing to release the code to a wider audience, it occurs to me that perhaps it may not be acceptable to the community of users to extend the 'conn_id' record. To be clear, the 'orig_u' element would be present within every log file that records the 'conn_id' record, such as http.log, ftp.log, dns.log, etc. However, the values are meaningful only for http.log. In the other log files, the 'orig_u' column would contain a dash '-' value indicating the value is unset. Design guidance: is it acceptable to redefine/extend the 'conn_id' record as described above? I appreciate your feedback. Thanks! Mark I. Fernandez -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160921/d1b11764/attachment-0001.html From seth at icir.org Thu Sep 22 07:37:07 2016 From: seth at icir.org (Seth Hall) Date: Thu, 22 Sep 2016 10:37:07 -0400 Subject: [Bro-Dev] ICAP Analyzer Design Guidance In-Reply-To: References: Message-ID: <2FAB39CC-0604-453D-93EB-0426406286C2@icir.org> > On Sep 21, 2016, at 5:03 PM, Fernandez, Mark I wrote: > > Design guidance: is it acceptable to redefine/extend the ?conn_id? record as described above? You probably don't want to extend the conn_id record. There are some cases where it can cause trouble doing lookups because the conn_id is used at a table index in a lot of places. Is there somewhere else you could stash the information that you need? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From seth at icir.org Thu Sep 22 07:45:42 2016 From: seth at icir.org (Seth Hall) Date: Thu, 22 Sep 2016 10:45:42 -0400 Subject: [Bro-Dev] bro-pkg -> bropkg In-Reply-To: <20160916205738.GI5321@icir.org> References: <2206E1C1-6B71-427F-89B8-3FA845604030@icir.org> <482461E8-B1EC-4A43-BE6A-28FD4F05C500@illinois.edu> <7E32F539-329F-4420-90B3-B52ACD062051@illinois.edu> <20160916205738.GI5321@icir.org> Message-ID: <2E0AC92D-1722-4CD6-B606-C30F6D95FAFA@icir.org> > On Sep 16, 2016, at 4:57 PM, Robin Sommer wrote: > > No strong preference either way on my end, but let's not touch bro-cut > please, that's used in too many scripts. Yeah, I think changing bro-cut is too far gone at this point. I was only asking about bro-pkg because it's so new and so few people have it installed. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From mfernandez at mitre.org Thu Sep 22 07:54:47 2016 From: mfernandez at mitre.org (Fernandez, Mark I) Date: Thu, 22 Sep 2016 14:54:47 +0000 Subject: [Bro-Dev] ICAP Analyzer Design Guidance In-Reply-To: <2FAB39CC-0604-453D-93EB-0426406286C2@icir.org> References: <2FAB39CC-0604-453D-93EB-0426406286C2@icir.org> Message-ID: Seth, >> Is there somewhere else you could stash the information that you need? Yes, I re-worked the script yesterday to redef/extend the HTTP::Info record and store the information there. But I notice it works differently than before, and I must do some extra effort to store it in the HTTP::Info record. Originally, in my 'icap_header' event handler within main.bro, I would check c?$http and create one if it did not exist yet for this connection. Within the same event, if the ICAP header is 'X-Authenticated-User', then I would copy that value into the modified 'conn_id' record within the 'c$http$id$orig_u' field. Easy peasy, the orig_u column would be added to every log file that prints the conn_id record, and that column would contain the correct value. But what I encountered yesterday when extending the HTTP::Info record to include the 'orig_u' field, it did not work so easily. Within the 'icap_header' event handler, I did everything the same except that I copied the value into 'c$http$orig_u' field (instead of 'c$http$id$orig_u'). However, it behaved differently: while the orig_u column would be added as the final column of the http.log (as expected), the value would be a dash '-', as if the value was unset. This was troubling me because I explicitly set the value within the 'icap_header' event handler. To remedy this, I had to create an event handler for 'http_request' and therein set the value of 'c$http$orig_u' accordingly. Fortunately, this worked, but I wonder why it did not work within 'icap_header', why the value was lost? Thanks! Mark I. Fernandez -----Original Message----- From: Seth Hall [mailto:seth at icir.org] Sent: Thursday, September 22, 2016 10:37 AM To: Fernandez, Mark I Cc: bro-dev at bro.org Subject: Re: [Bro-Dev] ICAP Analyzer Design Guidance > On Sep 21, 2016, at 5:03 PM, Fernandez, Mark I wrote: > > Design guidance: is it acceptable to redefine/extend the ?conn_id? record as described above? You probably don't want to extend the conn_id record. There are some cases where it can cause trouble doing lookups because the conn_id is used at a table index in a lot of places. Is there somewhere else you could stash the information that you need? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jsiwek at illinois.edu Thu Sep 22 08:39:08 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Thu, 22 Sep 2016 15:39:08 +0000 Subject: [Bro-Dev] Proposal: Operating the Bro package manager offline In-Reply-To: <20160921194546.GD37771@icir.org> References: <20160919232754.GD94635@icir.org> <20160921194546.GD37771@icir.org> Message-ID: <1E0F72C8-88D7-40F2-9792-FB647E8DCB78@illinois.edu> > On Sep 21, 2016, at 2:45 PM, Robin Sommer wrote: > > - Following Justin's suggestion, unbundling should by default not > replace everything currently installed; and instead offer a > "--replace" option if one wants that. What happens when --replace is not used and different version of a package that?s in the bundle is already installed? I think asking user whether it?s ok is the way to go, but how much attention to draw to it? E.g. $ bro-pkg unbundle mybundle.zip The following packages will be INSTALLED: foo (1.0.0) bar (1.0.0 -> 2.0.0) baz (4.0.0 -> 3.0.0) Proceed? [Y/n] On answering ?yes?, does it just go ahead or does it then ask for each individual package if it?s ok to overwrite it? - Jon From robin at icir.org Thu Sep 22 12:19:06 2016 From: robin at icir.org (Robin Sommer) Date: Thu, 22 Sep 2016 12:19:06 -0700 Subject: [Bro-Dev] Proposal: Operating the Bro package manager offline In-Reply-To: <1E0F72C8-88D7-40F2-9792-FB647E8DCB78@illinois.edu> References: <20160919232754.GD94635@icir.org> <20160921194546.GD37771@icir.org> <1E0F72C8-88D7-40F2-9792-FB647E8DCB78@illinois.edu> Message-ID: <20160922191906.GO87268@icir.org> On Thu, Sep 22, 2016 at 15:39 +0000, you wrote: > What happens when --replace is not used and different version of a > package that?s in the bundle is already installed? I think asking > user whether it?s ok is the way to go, Agree, plus a "--force" option once more to skip any questions for batch operation. > $ bro-pkg unbundle mybundle.zip > The following packages will be INSTALLED: > foo (1.0.0) > bar (1.0.0 -> 2.0.0) > baz (4.0.0 -> 3.0.0) > Proceed? [Y/n] > > On answering ?yes?, does it just go ahead or does it then ask for each individual package if it?s ok to overwrite it? Just a single "yes" sounds good to me, I would see it more as double-check to confirm the bundle is right. If it's not, one can (and should I argue) always rebuild the bundle with different packages. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From seth at icir.org Fri Sep 23 06:28:29 2016 From: seth at icir.org (Seth Hall) Date: Fri, 23 Sep 2016 09:28:29 -0400 Subject: [Bro-Dev] bro-pkg -> bropkg In-Reply-To: <2E0AC92D-1722-4CD6-B606-C30F6D95FAFA@icir.org> References: <2206E1C1-6B71-427F-89B8-3FA845604030@icir.org> <482461E8-B1EC-4A43-BE6A-28FD4F05C500@illinois.edu> <7E32F539-329F-4420-90B3-B52ACD062051@illinois.edu> <20160916205738.GI5321@icir.org> <2E0AC92D-1722-4CD6-B606-C30F6D95FAFA@icir.org> Message-ID: > On Sep 22, 2016, at 10:45 AM, Seth Hall wrote: > > Yeah, I think changing bro-cut is too far gone at this point. I was only asking about bro-pkg because it's so new and so few people have it installed. Quick follow up. I'm fine leaving things as-is. It's certainly easier and we can move on. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From slagell at illinois.edu Fri Sep 23 11:58:09 2016 From: slagell at illinois.edu (Slagell, Adam J) Date: Fri, 23 Sep 2016 18:58:09 +0000 Subject: [Bro-Dev] Blog post for bro-pkg Message-ID: Jon, Would you be up for writing a blog post introducing the Bro Package Manager and showing some examples of its use? We probably also want to state that these are unvetted contributions from the community. :Adam ------ Adam J. Slagell Chief Information Security Officer Director, Cybersecurity Division National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." From jsiwek at illinois.edu Sun Sep 25 11:24:10 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Sun, 25 Sep 2016 18:24:10 +0000 Subject: [Bro-Dev] Blog post for bro-pkg In-Reply-To: References: Message-ID: Yeah, I can write a blog post. - Jon > On Sep 23, 2016, at 1:58 PM, Slagell, Adam J wrote: > > Jon, > > Would you be up for writing a blog post introducing the Bro Package Manager and showing some examples of its use? We probably also want to state that these are unvetted contributions from the community. > > :Adam > > > > ------ > > Adam J. Slagell > Chief Information Security Officer > Director, Cybersecurity Division > National Center for Supercomputing Applications > University of Illinois at Urbana-Champaign > www.slagell.info > > "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." > > > > > > > > From jsiwek at illinois.edu Thu Sep 29 13:49:51 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Thu, 29 Sep 2016 20:49:51 +0000 Subject: [Bro-Dev] Proposal: Operating the Bro package manager offline In-Reply-To: <20160922191906.GO87268@icir.org> References: <20160919232754.GD94635@icir.org> <20160921194546.GD37771@icir.org> <1E0F72C8-88D7-40F2-9792-FB647E8DCB78@illinois.edu> <20160922191906.GO87268@icir.org> Message-ID: ?bundle? and ?unbundle? commands are now available in bro-pkg 0.7 and should work as described earlier in the thread. - Jon From robin at icir.org Fri Sep 30 07:33:35 2016 From: robin at icir.org (Robin Sommer) Date: Fri, 30 Sep 2016 07:33:35 -0700 Subject: [Bro-Dev] Proposal: Operating the Bro package manager offline In-Reply-To: References: <20160919232754.GD94635@icir.org> <20160921194546.GD37771@icir.org> <1E0F72C8-88D7-40F2-9792-FB647E8DCB78@illinois.edu> <20160922191906.GO87268@icir.org> Message-ID: <20160930143335.GB46186@icir.org> On Thu, Sep 29, 2016 at 20:49 +0000, you wrote: > ?bundle? and ?unbundle? commands are now available in bro-pkg 0.7 and > should work as described earlier in the thread. Great, many thanks! I'll give it a try soon. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From mfernandez at mitre.org Fri Sep 30 11:42:58 2016 From: mfernandez at mitre.org (Fernandez, Mark I) Date: Fri, 30 Sep 2016 18:42:58 +0000 Subject: [Bro-Dev] ICAP Analyzer: BinPAC vs Plugin :: RegEx Issues Message-ID: In support of submitting the ICAP Analyzer as a Bro Package, I am porting the ICAP Analyzer to build as a dynamic Plugin. Originally, I inserted the ICAP Analyzer straight into the source code tree, under /src/analyzer/protocol/icap, and compiled it as part of the Bro core. But in an effort to make it easier for others to integrate into their existing Bro instantiations, I am making the effort to make it a stand-alone Plugin instead... but the BinPAC parser is not working when I run it as a Plugin. The Plugin builds and installs without error, and I verify that the Plugin is enabled and that my ICAP main.bro script is loaded, but it is not producing any ICAP or HTTP related output: (a) It appears that the parser is not recognizing the ICAP Request messages whatsoever. (b) It starts to parse the ICAP Response messages; but it breaks mid-way thru the packet. I think the problem is within the BinPAC files where I use regular expressions to define a data element within the ICAP packet structures/records. In the ICAP Request message, the very first element is a regex pattern, so that's why it fails to parse these packets at all. In the ICAP Response message, it parses the first element correctly, but then it bombs on the second element, which is a regex pattern. In the BinPAC help/reference document, it contains a section titled, "Running Binpac-Generated Analyzer Standalone" [https://www.bro.org/sphinx/components/binpac/README.html#running-binpac-generated-analyzer-standalone], which states that to run binpac-generated code independent of Bro, the regex library must be substituted... I presume the stand-alone guidance applies to the Plugin? It must because I did not have this trouble when I built the analyzer straight into the Bro core. The regex library guidance says I need to include three header files: RE.h, bro-dummy.h, and binpac_pcre.h. You provide sample code for each file. Am I to copy-n-paste the sample code directly into my Plugin source code as three new headers files? Or do these three files exist elsewhere in the Bro source? I can find "RE.h" in the source (/src/RE.h). And I can find "binpac_regex.h" in the source (/aux/binpac/lib/binpac_regex.h), which seems similar, but I cannot find "binpac_pcre.h" nor "bro_dummy.h" anywhere. I need a little bit of advice... or a lot of advice :) Can I use RE.h and binpac_regex.h that exist in the Bro 2.4.1 distro? Or do I need to create the three header files and paste the sample code verbatim? Thanks! Mark Mark I. Fernandez MITRE Corporation Email: mfernandez at mitre.org MITRE is a not-for-profit corporation that operates several Federally Funded Research and Development Centers (FFRDCs) in the interests of the US Government. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160930/4d65cd9f/attachment-0001.html