From balint.martina at gmail.com Tue Nov 1 05:49:18 2016 From: balint.martina at gmail.com (Martina Balintova) Date: Tue, 1 Nov 2016 13:49:18 +0100 Subject: [Bro-Dev] Logs columm names Message-ID: Hi, not sure if I missed this feature in documentation, but is it possible to assign different name to columns that are logged? I would like to change name of column, but I do not want to go on searching for all places where variable (also used as a current name of the column) is used and rename it. I just want to change name of column that gets to logs. If this feature is not there yet, could you pls consider adding it? maybe using &log eg: type Info: record { variable_name_not_presentable : string &log=PresentableName; } Martina -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20161101/9910cdbe/attachment.html From mfernandez at mitre.org Tue Nov 1 06:44:43 2016 From: mfernandez at mitre.org (Fernandez, Mark I) Date: Tue, 1 Nov 2016 13:44:43 +0000 Subject: [Bro-Dev] Bro Debug Mode :: Assert Failed in HTTP.cc Line 156 Message-ID: Bro Dev Team, While trying to translate the ICAP Analyzer into a Bro Plugin, I discovered that my ICAP Analyzer fails an assertion in HTTP.cc Line 156. I discovered it only recently because last week I compiled Bro in debug mode for the first time in order to troubleshoot the Plugin. (Fact: I never compiled Bro in debug mode when originally developing the ICAP Analyzer last year, didn't need to do so at the time.) Shown below is a sample of the code trace, starting in the ICAP Analyzer and transiting into the HTTP Analyzer. Source File: icap-analyzer-http.pac Multiple calls to HTTP_Analyzer::DeliverStream() Source File: HTTP.cc Line 875, void HTTP_Analyzer::DeliverStream(int len, const u_char* data, bool is_orig)) ... Line 1018, case EXPECT_REPLY_MESSAGE: Line 1019, reply_message->Deliver(len, line, 1); Line 78, void HTTP_Entity::Deliver(int len, const char* data, int trailing_CRLF) ... Line 133, case EXPECT_CHUNK_DATA: Line 134, ASSERT(! trailing_CRLF); As you can see, Line 1019 passes '1' as the value for trailing_CRLF and then Line 134 complains if the value is '1', causing Bro to abort. When compiled in release mode, this failed assert does not affect operations. The ICAP Analyzer is able to inject the ICAP payload into the HTTP Analyzer, and it functions as expected. However, in debug mode, the failed assert cause Bro to abort. I thought this was unusual, and I wanted to report it. Cheers! Mark Mark I. Fernandez MITRE Corporation Email: mfernandez at mitre.org MITRE is a not-for-profit corporation that operates several Federally Funded Research and Development Centers (FFRDCs) in the interests of the US Government. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20161101/1b94898b/attachment-0001.html From mfernandez at mitre.org Tue Nov 1 06:49:23 2016 From: mfernandez at mitre.org (Fernandez, Mark I) Date: Tue, 1 Nov 2016 13:49:23 +0000 Subject: [Bro-Dev] Bro Debug Mode :: Assert Failed in HTTP.cc Line 156 In-Reply-To: References: Message-ID: BIG OOPS. I mis-matched the code snippet in my original email. Correct snippet for Line 156 is shown below: Source File: icap-analyzer-http.pac Multiple calls to HTTP_Analyzer::DeliverStream() Source File: HTTP.cc Line 875, void HTTP_Analyzer::DeliverStream(int len, const u_char* data, bool is_orig)) ... Line 1018, case EXPECT_REPLY_MESSAGE: Line 1019, reply_message->Deliver(len, line, 1); Line 78, void HTTP_Entity::Deliver(int len, const char* data, int trailing_CRLF) ... Line 154, else if( content_length >= 0 ) Line 155, { Line 156, ASSERT(! trailing_CRLF); As you can see, Line 1019 passes '1' as the value for trailing_CRLF and then Line 156 complains if the value is '1', causing Bro to abort. Mark From: Fernandez, Mark I Sent: Tuesday, November 01, 2016 9:45 AM To: 'bro-dev at bro.org' Subject: Bro Debug Mode :: Assert Failed in HTTP.cc Line 156 Bro Dev Team, While trying to translate the ICAP Analyzer into a Bro Plugin, I discovered that my ICAP Analyzer fails an assertion in HTTP.cc Line 156. I discovered it only recently because last week I compiled Bro in debug mode for the first time in order to troubleshoot the Plugin. (Fact: I never compiled Bro in debug mode when originally developing the ICAP Analyzer last year, didn't need to do so at the time.) Shown below is a sample of the code trace, starting in the ICAP Analyzer and transiting into the HTTP Analyzer. Source File: icap-analyzer-http.pac Multiple calls to HTTP_Analyzer::DeliverStream() Source File: HTTP.cc Line 875, void HTTP_Analyzer::DeliverStream(int len, const u_char* data, bool is_orig)) ... Line 1018, case EXPECT_REPLY_MESSAGE: Line 1019, reply_message->Deliver(len, line, 1); Line 78, void HTTP_Entity::Deliver(int len, const char* data, int trailing_CRLF) ... Line 133, case EXPECT_CHUNK_DATA: Line 134, ASSERT(! trailing_CRLF); As you can see, Line 1019 passes '1' as the value for trailing_CRLF and then Line 134 complains if the value is '1', causing Bro to abort. When compiled in release mode, this failed assert does not affect operations. The ICAP Analyzer is able to inject the ICAP payload into the HTTP Analyzer, and it functions as expected. However, in debug mode, the failed assert cause Bro to abort. I thought this was unusual, and I wanted to report it. Cheers! Mark Mark I. Fernandez MITRE Corporation Email: mfernandez at mitre.org MITRE is a not-for-profit corporation that operates several Federally Funded Research and Development Centers (FFRDCs) in the interests of the US Government. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20161101/c1ad4ae2/attachment-0001.html From robin at icir.org Tue Nov 1 09:37:56 2016 From: robin at icir.org (Robin Sommer) Date: Tue, 1 Nov 2016 09:37:56 -0700 Subject: [Bro-Dev] Logs columm names In-Reply-To: References: Message-ID: <20161101163755.GD21981@icir.org> On Tue, Nov 01, 2016 at 13:49 +0100, you wrote: > not sure if I missed this feature in documentation, but is it possible to > assign different name to columns that are logged? Bro 2.5 supports this (currently in beta): https://github.com/bro/bro/commit/b28801ce9508250594a8e3c6483f3176483d790c Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From balint.martina at gmail.com Mon Nov 7 06:33:50 2016 From: balint.martina at gmail.com (Martina Balintova) Date: Mon, 7 Nov 2016 15:33:50 +0100 Subject: [Bro-Dev] Array parsing and indexing In-Reply-To: References: Message-ID: Hi, could you pls help me with one parser? I am trying to parse protocol that can contain list of chained commands coming in one packet. Lets say that structure looks like this: type Request(header: Head) = record { cmd_count : uint8; reserved : uint8; cmds : uint16[cmd_count]; params: Request_Array(cmds, header)[cmd_count]; }; cmd_count contains number of commands that are chained, cmds are command IDENTIFIERS listed one after another up to cmd_count and params are parameters and argument for individual commands. Individual params have different types depending on command indentifiers listed in cmds array. Could you pls help me define these two types? how should 'params' look like? How can I pass index into cmds array? type Request_Array(index: uint16, header: Head) = case index of { INDEX_00 -> open : request_open(header); INDEX_01 -> find : request_find(header); default -> empty : bytestring &restofdata &transient; }; example of packet: { cmd_count : 3 _reserved_ cmds: 0, 0, 1 params[0]: .... params[0]: .... params[1]:..... } Any suggestions? Martina -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20161107/25c3cc29/attachment.html From jsiwek at illinois.edu Fri Nov 18 12:57:49 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Fri, 18 Nov 2016 20:57:49 +0000 Subject: [Bro-Dev] Package manager meta data In-Reply-To: <1303E27E-FC8D-4FA5-AD68-DA64FC401F84@illinois.edu> References: <20161027232819.GD20933@icir.org> <9A7795D3-3C47-475B-AEB1-2CEDF9BB7AE4@illinois.edu> <12c859cf-b998-e726-c852-1caa32333e16@gmail.com> <536BD67A-7A29-4B3F-9ABB-38DA079C53F3@illinois.edu> <1760cb97-de75-1cf7-b42e-899e4688418f@gmail.com> <1303E27E-FC8D-4FA5-AD68-DA64FC401F84@illinois.edu> Message-ID: <63694BF3-3E9D-4419-B31D-B96C9E980364@illinois.edu> Here?s a summary of the set of changes I plan to make related to package metadata, let me know if there?s objections or alternate ideas: 1) remove mentions of ?version' field from bro-pkg.meta as versioning is done entirely by git tags 2) packages should now put ?description? and ?tags' fields into bro-pkg.meta so they are fully self-describing 3) change "info" command to first check if package is already installed before trying to download it. This allows metadata to be displayed when working offline. 4) new command/flag `bro-pkg refresh --aggregate-metadata `: crawls a package source?s index files and for each package take the master branch?s bro-pkg.meta and aggregate it into a single file, aggregate.meta, at the top-level of the package source 5) new command/flag `bro-pkg refresh --aggregate-metadata ?push`: same as (4), but also commit/push changes to remote repo 6) change structure of package source index files: bro-pkg.index will now contain just a simple list of package URLs: https://github.com/sethhall/credit-card-exposure https://github.com/sethhall/ssn-exposure https://github.com/sethhall/domain-tld Aggregated metadata from packages in the source all lives within a single file at the top-level named ?aggregate.meta?. Only metadata from current master branches of packages is included ? I don?t think it?s that useful to collect metadata for each release version and doing that even makes it harder for package authors to change the way users discover their package. - Jon From robin at icir.org Sun Nov 20 11:57:38 2016 From: robin at icir.org (Robin Sommer) Date: Sun, 20 Nov 2016 11:57:38 -0800 Subject: [Bro-Dev] Package manager meta data In-Reply-To: <63694BF3-3E9D-4419-B31D-B96C9E980364@illinois.edu> References: <9A7795D3-3C47-475B-AEB1-2CEDF9BB7AE4@illinois.edu> <12c859cf-b998-e726-c852-1caa32333e16@gmail.com> <536BD67A-7A29-4B3F-9ABB-38DA079C53F3@illinois.edu> <1760cb97-de75-1cf7-b42e-899e4688418f@gmail.com> <1303E27E-FC8D-4FA5-AD68-DA64FC401F84@illinois.edu> <63694BF3-3E9D-4419-B31D-B96C9E980364@illinois.edu> Message-ID: <20161120195738.GE7553@icir.org> On Fri, Nov 18, 2016 at 20:57 +0000, you wrote: > Here?s a summary of the set of changes I plan to make related to > package metadata, let me know if there?s objections or alternate > ideas: Sounds all good to me. That should make the Python API easier to use for accessing meta information, too. Some questions for my understanding: > remove mentions of ?version' field from bro-pkg.meta as versioning is > done entirely by git tags What will be the relationship between version tags and the master branch? Is master just another version I can select to install? > 4) new command/flag `bro-pkg refresh --aggregate-metadata source>`: A shortcut "bro-pkg refresh" might be useful here that simply does that for all configured sources. Also, what if I update the meta information locally, but then later the remote repository updates its version as well, so that mine gets stale. Would the newer remote version replace the local version? > single file at the top-level named ?aggregate.meta?. Only metadata > from current master branches of packages is included Depending on the answer to the version question above, an alternative could be using the most recent version (i.e., the commit that that version tag refers to). That way people could control a bit better when meta data gets updated (it's when they set a new version). It would still fallback to master if no version tag is set. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jsiwek at illinois.edu Mon Nov 21 07:53:27 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Mon, 21 Nov 2016 15:53:27 +0000 Subject: [Bro-Dev] Package manager meta data In-Reply-To: <20161120195738.GE7553@icir.org> References: <9A7795D3-3C47-475B-AEB1-2CEDF9BB7AE4@illinois.edu> <12c859cf-b998-e726-c852-1caa32333e16@gmail.com> <536BD67A-7A29-4B3F-9ABB-38DA079C53F3@illinois.edu> <1760cb97-de75-1cf7-b42e-899e4688418f@gmail.com> <1303E27E-FC8D-4FA5-AD68-DA64FC401F84@illinois.edu> <63694BF3-3E9D-4419-B31D-B96C9E980364@illinois.edu> <20161120195738.GE7553@icir.org> Message-ID: <53D56399-E227-488D-81E6-441625ADD0A9@illinois.edu> > On Nov 20, 2016, at 1:57 PM, Robin Sommer wrote: > > What will be the relationship between version tags and the master > branch? Is master just another version I can select to install? Yes, any tag or branch can be selected with `bro-pkg install ?version `. When selecting a version, `bro-pkg update` only considers newer x.y.z release tags for that package. When selecting a branch, it tracks updates along that branch. >> 4) new command/flag `bro-pkg refresh --aggregate-metadata > source>`: > > A shortcut "bro-pkg refresh" might be useful here that simply does > that for all configured sources. I?ll make ?aggregate flag operate on all sources then add an optional "?source ? flag for those that just want to operate on a single one. `bro-pkg refresh` without any flags is currently like a `git pull` from the remote source and what most users will be doing. Likely only the operators of sources will want to ?aggregate (if the package ecosystem becomes large, this operation could take a while to complete). > Also, what if I update the meta information locally, but then later > the remote repository updates its version as well, so that mine gets > stale. Would the newer remote version replace the local version? Yes, newer data from the remote source overwrites local changes. >> single file at the top-level named ?aggregate.meta?. Only metadata >> from current master branches of packages is included > > Depending on the answer to the version question above, an alternative > could be using the most recent version (i.e., the commit that that > version tag refers to). That way people could control a bit better > when meta data gets updated (it's when they set a new version). It > would still fallback to master if no version tag is set. Sounds good, I?ll have it do that. - Jon From robin at icir.org Mon Nov 21 08:38:12 2016 From: robin at icir.org (Robin Sommer) Date: Mon, 21 Nov 2016 08:38:12 -0800 Subject: [Bro-Dev] Package manager meta data In-Reply-To: <53D56399-E227-488D-81E6-441625ADD0A9@illinois.edu> References: <12c859cf-b998-e726-c852-1caa32333e16@gmail.com> <536BD67A-7A29-4B3F-9ABB-38DA079C53F3@illinois.edu> <1760cb97-de75-1cf7-b42e-899e4688418f@gmail.com> <1303E27E-FC8D-4FA5-AD68-DA64FC401F84@illinois.edu> <63694BF3-3E9D-4419-B31D-B96C9E980364@illinois.edu> <20161120195738.GE7553@icir.org> <53D56399-E227-488D-81E6-441625ADD0A9@illinois.edu> Message-ID: <20161121163812.GL7553@icir.org> On Mon, Nov 21, 2016 at 15:53 +0000, you wrote: > When selecting a version, `bro-pkg update` only considers newer x.y.z > release tags for that package. When selecting a branch, it tracks > updates along that branch. Ok, sounds like I should think about it not as "master" then, but as "HEAD on the master branch". So let's say the first time I install a package, there're tags v1.0 and v1.1 on master, but master's HEAD is already a bit ahead, without a new version set yet. What gets installed when I don't specify a version? 1.1? And what does update do when (1) later a new version got set, and (2) master then moves beyond that version? It would be good to spell this all out clearly in the documentation, including in the quick start / examples, so that people understand how to use the versioning effectively. > I?ll make ?aggregate flag operate on all sources then add an optional > "?source ? flag for those that just want to operate on a > single one. Makes sense. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jsiwek at illinois.edu Mon Nov 21 10:24:21 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Mon, 21 Nov 2016 18:24:21 +0000 Subject: [Bro-Dev] Package manager meta data In-Reply-To: <20161121163812.GL7553@icir.org> References: <12c859cf-b998-e726-c852-1caa32333e16@gmail.com> <536BD67A-7A29-4B3F-9ABB-38DA079C53F3@illinois.edu> <1760cb97-de75-1cf7-b42e-899e4688418f@gmail.com> <1303E27E-FC8D-4FA5-AD68-DA64FC401F84@illinois.edu> <63694BF3-3E9D-4419-B31D-B96C9E980364@illinois.edu> <20161120195738.GE7553@icir.org> <53D56399-E227-488D-81E6-441625ADD0A9@illinois.edu> <20161121163812.GL7553@icir.org> Message-ID: <16D5375C-632E-4FE6-A055-379D40A00B25@illinois.edu> > On Nov 21, 2016, at 10:38 AM, Robin Sommer wrote: > > On Mon, Nov 21, 2016 at 15:53 +0000, you wrote: > >> When selecting a version, `bro-pkg update` only considers newer x.y.z >> release tags for that package. When selecting a branch, it tracks >> updates along that branch. > > Ok, sounds like I should think about it not as "master" then, but as > "HEAD on the master branch?. For packages tracking a git branch, bro-pkg will update to the latest commit at the end of the branch. Maybe I?m getting git terminology wrong, but I thought ?master? is always a reference to the latest commit at the end of the branch and HEAD usually means whatever commit is locally checked out (maybe not latest commit). So thinking of it as ?master? sounds right to me, but whatever you think of as the ?end of the branch? is what bro-pkg is tracking. > So let's say the first time I install a > package, there're tags v1.0 and v1.1 on master, but master's HEAD is > already a bit ahead, without a new version set yet. What gets > installed when I don't specify a version? 1.1? 1.1 is installed by default. (rationale is to give preference to latest stable release over dev versions) > And what does update do > when (1) later a new version got set, and (2) master then moves beyond > that version? (1) it installs the newer version tag (?newer? meaning the version number is greater than before) (2) it doesn?t care what gets added on master, the local install isn?t updated In other words, if you install a specific release tag of a package (by default or by choice), you only receive stable release updates. If you install a package?s branch (by default or by choice), updates will be tracking the latest commit at the end of that branch. > It would be good to spell this all out clearly in the documentation, > including in the quick start / examples, so that people understand how > to use the versioning effectively. Just added another section also linked to it in quickstart: http://bro-package-manager.readthedocs.io/en/latest/package.html#package-versioning The upgrade command docs also have a brief description of how it works. - Jon From robin at icir.org Mon Nov 21 10:33:01 2016 From: robin at icir.org (Robin Sommer) Date: Mon, 21 Nov 2016 10:33:01 -0800 Subject: [Bro-Dev] Package manager meta data In-Reply-To: <16D5375C-632E-4FE6-A055-379D40A00B25@illinois.edu> References: <536BD67A-7A29-4B3F-9ABB-38DA079C53F3@illinois.edu> <1760cb97-de75-1cf7-b42e-899e4688418f@gmail.com> <1303E27E-FC8D-4FA5-AD68-DA64FC401F84@illinois.edu> <63694BF3-3E9D-4419-B31D-B96C9E980364@illinois.edu> <20161120195738.GE7553@icir.org> <53D56399-E227-488D-81E6-441625ADD0A9@illinois.edu> <20161121163812.GL7553@icir.org> <16D5375C-632E-4FE6-A055-379D40A00B25@illinois.edu> Message-ID: <20161121183301.GS7553@icir.org> Thanks, that and the new text help, I think I got it now. That all makes sense. Maybe add to the text that tags are not pushed by default by git, so one shouldn't forget "git push --tags". Robin On Mon, Nov 21, 2016 at 18:24 +0000, you wrote: > > > On Nov 21, 2016, at 10:38 AM, Robin Sommer wrote: > > > > On Mon, Nov 21, 2016 at 15:53 +0000, you wrote: > > > >> When selecting a version, `bro-pkg update` only considers newer x.y.z > >> release tags for that package. When selecting a branch, it tracks > >> updates along that branch. > > > > Ok, sounds like I should think about it not as "master" then, but as > > "HEAD on the master branch?. > > For packages tracking a git branch, bro-pkg will update to the latest commit at the end of the branch. > > Maybe I?m getting git terminology wrong, but I thought ?master? is always a reference to the latest commit at the end of the branch and HEAD usually means whatever commit is locally checked out (maybe not latest commit). So thinking of it as ?master? sounds right to me, but whatever you think of as the ?end of the branch? is what bro-pkg is tracking. > > > So let's say the first time I install a > > package, there're tags v1.0 and v1.1 on master, but master's HEAD is > > already a bit ahead, without a new version set yet. What gets > > installed when I don't specify a version? 1.1? > > 1.1 is installed by default. (rationale is to give preference to latest stable release over dev versions) > > > And what does update do > > when (1) later a new version got set, and (2) master then moves beyond > > that version? > > (1) it installs the newer version tag (?newer? meaning the version number is greater than before) > (2) it doesn?t care what gets added on master, the local install isn?t updated > > In other words, if you install a specific release tag of a package (by default or by choice), you only receive stable release updates. > > If you install a package?s branch (by default or by choice), updates will be tracking the latest commit at the end of that branch. > > > It would be good to spell this all out clearly in the documentation, > > including in the quick start / examples, so that people understand how > > to use the versioning effectively. > > Just added another section also linked to it in quickstart: > > http://bro-package-manager.readthedocs.io/en/latest/package.html#package-versioning > > The upgrade command docs also have a brief description of how it works. > > - Jon > -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From scampbell at lbl.gov Mon Nov 28 08:11:08 2016 From: scampbell at lbl.gov (Scott Campbell) Date: Mon, 28 Nov 2016 11:11:08 -0500 Subject: [Bro-Dev] Packet Brick question(s) Message-ID: <551d1832-bf3a-8847-025e-8f4000363f97@lbl.gov> I have been investigating the use of Packet Bricks/netmap as a replacement for pf_ring on linux, but have a few questions. (1) Is there any documentation except for the git page and the scripts themselves? The script comments are nice and useful, but at times the syntax is rather opaque. (2) Following directions and mailing list recommendations, I have a working version which reads from a heavily loaded 10G ixgbe interface and splits the traffic into 4 netmap interfaces. The script looks like: utilObj:enable_nmpipes() pe = PktEngine.new("e0", 1024, 8) lb = Brick.new("LoadBalancer", 2) lb:connect_input("eth6") lb:connect_output("eth6{0", "eth6{1", "eth6{2", "eth6{3") pe:link(lb) pe:start() where eth6 is the data source interface. Script output looks like: > [root at xdev-w4 PB_INSTALL]# sbin/bricks -f etc/bricks-scripts/startup-one-thread.lua > [ pmain(): line 466] Executing etc/bricks-scripts/startup-one-thread.lua > [print_version(): line 348] BRICKS Version 0.5-beta > bricks> utilObj:enable_nmpipes() > bricks> pe = PktEngine.new("e0", 1024, 8) > bricks> lb = Brick.new("LoadBalancer", 2) > bricks> lb:connect_input("eth6") > bricks> lb:connect_output("eth6{0", "eth6{1", "eth6{2", "eth6{3") > bricks> pe:link(lb) > [ lb_init(): line 66] Adding brick eth6{0 to the engine > [ promisc(): line 96] Interface eth6 is already set to promiscuous mode > 970.328612 nm_open [444] overriding ARG3 0 > 970.328631 nm_open [457] overriding ifname eth6 ringid 0x0 flags 0x1 > [netmap_link_iface(): line 183] Wait for 2 secs for phy reset > [brick_link(): line 113] Linking e0 with link eth6 with batch size: 512 and qid: -1 > [netmap_create_channel(): line 746] brick: 0xfac090, local_desc: 0xfac780 > 972.343050 nm_open [444] overriding ARG3 0 > [netmap_create_channel(): line 781] zerocopy for eth6 --> eth6{0 (index: 0) enabled > [netmap_create_channel(): line 786] Created netmap:eth6{0 interface > [netmap_create_channel(): line 746] brick: 0xfac090, local_desc: 0xfac780 > 972.343600 nm_open [444] overriding ARG3 0 > [netmap_create_channel(): line 781] zerocopy for eth6 --> eth6{1 (index: 1) enabled > [netmap_create_channel(): line 786] Created netmap:eth6{1 interface > [netmap_create_channel(): line 746] brick: 0xfac090, local_desc: 0xfac780 > 972.344200 nm_open [444] overriding ARG3 0 > [netmap_create_channel(): line 781] zerocopy for eth6 --> eth6{2 (index: 2) enabled > [netmap_create_channel(): line 786] Created netmap:eth6{2 interface > [netmap_create_channel(): line 746] brick: 0xfac090, local_desc: 0xfac780 > 972.344696 nm_open [444] overriding ARG3 0 > [netmap_create_channel(): line 781] zerocopy for eth6 --> eth6{3 (index: 3) enabled > [netmap_create_channel(): line 786] Created netmap:eth6{3 interface > bricks> pe:start() and the related dmesg data is: > dmesg: > ixgbe 0000:81:00.0: eth6: detected SFP+: 5 > ixgbe 0000:81:00.0: eth6: NIC Link is Up 10 Gbps, Flow Control: RX/TX > 494.566450 [ 131] ixgbe_netmap_configure_srrctl bufsz: 2048 srrctl: 2 > ixgbe 0000:81:00.0: eth6: detected SFP+: 5 > ixgbe 0000:81:00.0: eth6: NIC Link is Up 10 Gbps, Flow Control: RX/TX > 496.743920 [ 320] netmap_pipe_krings_create ffff880876731a00: case 1, create both ends > 496.744464 [ 320] netmap_pipe_krings_create ffff880876731000: case 1, create both ends > 496.745026 [ 320] netmap_pipe_krings_create ffff880878fcb600: case 1, create both ends > 496.745520 [ 320] netmap_pipe_krings_create ffff880875e06c00: case 1, create both ends > Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-netmap instead > Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-netmap instead When I run the "./pkt-gen -i netmap:eth6}0 -f rx" command, I am seeing only a tiny fraction of the expected traffic: > [root at xdev-w4 bin]# ./pkt-gen -i netmap:eth6}0 -f rx > 007.093750 main [2552] interface is netmap:eth6}0 > 007.093855 main [2675] running on 1 cpus (have 32) > 007.094406 extract_ip_range [465] range is 10.0.0.1:1234 to 10.0.0.1:1234 > 007.094418 extract_ip_range [465] range is 10.1.0.1:1234 to 10.1.0.1:1234 > 007.094481 main [2770] mapped 334980KB at 0x7f325200d000 > Receiving from netmap:eth6}0: 1 queues, 1 threads and 1 cpus. > 007.094525 start_threads [2235] Wait 2 secs for phy reset > 009.094811 start_threads [2237] Ready... > 009.094927 receiver_body [1638] reading from netmap:eth6}0 fd 3 main_fd 3 > 010.095975 main_thread [2325] 3.573 Kpps (3.577 Kpkts 2.151 Mbps in 1001085 usec) 511.00 avg_batch 0 min_space > 011.097211 main_thread [2325] 2.552 Kpps (2.555 Kpkts 1.643 Mbps in 1001237 usec) 511.00 avg_batch 1 min_space > 012.098314 main_thread [2325] 3.063 Kpps (3.066 Kpkts 1.981 Mbps in 1001103 usec) 511.00 avg_batch 1 min_space ... > ^C021.032505 sigint_h [512] received control-C on thread 0x7f326672f700 > 021.032531 main_thread [2325] 2.762 Kpps (2.555 Kpkts 1.306 Mbps in 925126 usec) 511.00 avg_batch 1 min_space > 022.033620 main_thread [2325] 510.000 pps (511.000 pkts 306.248 Kbps in 1001087 usec) 511.00 avg_batch 1 min_space > Received 33726 packets 2876632 bytes 66 events 85 bytes each in 12.02 seconds. > Speed: 2.806 Kpps Bandwidth: 1.915 Mbps (raw 2.453 Mbps). Average batch: 511.00 pkts Running all 4 netmap interfaces provides about the same volume of data (which is in total ~ 1% of what I would expect). The cpu usage of the bricks command is ~ 15-20% regardless of running pkt-gen. What can I do differently to get better performance? (3) Accessing the interfaces - as far as actually using the interfaces with bro or tcpdump I am somewhat at a loss. I installed netmap-libpcap version 1.6.0-PRE-GIT_2016_11_26 and compiled tcpdump: [root at xdev-w4 tcpdump-4.6.2]# ./tcpdump --version tcpdump version 4.6.2 libpcap version 1.6.0-PRE-GIT_2016_11_26 OpenSSL 1.0.1e-fips 11 Feb 2013 which is the recommended version per the information on the packet bricks git README. I am unable to get either the native or the netmap-libpcap version of tcpdump to recognize the interfaces: [root at xdev-w4 tcpdump-4.6.2]# ./tcpdump -n -i netmap:eth6{0 tcpdump: netmap:eth6{0: No such device exists (SIOCGIFHWADDR: No such device) The ifconfig info for the interface looks like: > eth6 Link encap:Ethernet HWaddr 00:1B:21:9D:95:EA > UP BROADCAST RUNNING PROMISC MULTICAST MTU:9000 Metric:1 > RX packets:39463364964 errors:0 dropped:72437 overruns:0 frame:0 > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:56963040769452 (51.8 TiB) TX bytes:0 (0.0 b) At this point I am wondering where to go next. It would be great to use PB instead of pf_ring, but I will need some help to get there. many thanks! scott -- "The enemy knows the system." ? Claude Shannon From asharma at lbl.gov Mon Nov 28 08:37:46 2016 From: asharma at lbl.gov (Aashish Sharma) Date: Mon, 28 Nov 2016 08:37:46 -0800 Subject: [Bro-Dev] Packet Brick question(s) In-Reply-To: <551d1832-bf3a-8847-025e-8f4000363f97@lbl.gov> References: <551d1832-bf3a-8847-025e-8f4000363f97@lbl.gov> Message-ID: <20161128163744.GD85419@mac-799.local> Scott, I was using the following script when I was playing with the packet-bricks Dec last year: utilObj = dofile("scripts/utils.lua") utilObj:enable_nmpipes() pe = PktEngine.new("e0") lb = Brick.new("LoadBalancer", 2) lb:connect_input("ix0") lb:connect_output("ix0{1", "ix0{2", "ix0{3", "ix0{4", "ix0{5", "ix0{6", "ix0{7", "ix0{8", "ix0{9", "ix0{10", "ix0{11", "ix0{12", "ix0{13", "ix0{14", "ix0{15", "ix0{16", "ix0{17", "ix0{18", "ix0{19", "ix0{20" ) pe:link(lb) pe:start() pe:show_stats() This was on FreeBSD. > When I run the "./pkt-gen -i netmap:eth6}0 -f rx" command, I am seeing > only a tiny fraction of the expected traffic: Uncharted terretory for me. I was actually comparing bro logs generated from packet-bricks with the ones on other clusters and bricks vs others seemed reasonably comparable when I last left playing with this. > What can I do differently to get better performance? Others can elaborate more since I haven't tested this at all but I hear netmap support in linux is quite stable now - that might be something to try. Aashish On Mon, Nov 28, 2016 at 11:11:08AM -0500, Scott Campbell wrote: > I have been investigating the use of Packet Bricks/netmap as a > replacement for pf_ring on linux, but have a few questions. > > (1) Is there any documentation except for the git page and the scripts > themselves? The script comments are nice and useful, but at times the > syntax is rather opaque. > > (2) Following directions and mailing list recommendations, I have a > working version which reads from a heavily loaded 10G ixgbe interface > and splits the traffic into 4 netmap interfaces. The script looks like: > > utilObj:enable_nmpipes() > pe = PktEngine.new("e0", 1024, 8) > lb = Brick.new("LoadBalancer", 2) > lb:connect_input("eth6") > lb:connect_output("eth6{0", "eth6{1", "eth6{2", "eth6{3") > pe:link(lb) > pe:start() > > where eth6 is the data source interface. > > Script output looks like: > > [root at xdev-w4 PB_INSTALL]# sbin/bricks -f etc/bricks-scripts/startup-one-thread.lua > > [ pmain(): line 466] Executing etc/bricks-scripts/startup-one-thread.lua > > [print_version(): line 348] BRICKS Version 0.5-beta > > bricks> utilObj:enable_nmpipes() > > bricks> pe = PktEngine.new("e0", 1024, 8) > > bricks> lb = Brick.new("LoadBalancer", 2) > > bricks> lb:connect_input("eth6") > > bricks> lb:connect_output("eth6{0", "eth6{1", "eth6{2", "eth6{3") > > bricks> pe:link(lb) > > [ lb_init(): line 66] Adding brick eth6{0 to the engine > > [ promisc(): line 96] Interface eth6 is already set to promiscuous mode > > 970.328612 nm_open [444] overriding ARG3 0 > > 970.328631 nm_open [457] overriding ifname eth6 ringid 0x0 flags 0x1 > > [netmap_link_iface(): line 183] Wait for 2 secs for phy reset > > [brick_link(): line 113] Linking e0 with link eth6 with batch size: 512 and qid: -1 > > [netmap_create_channel(): line 746] brick: 0xfac090, local_desc: 0xfac780 > > 972.343050 nm_open [444] overriding ARG3 0 > > [netmap_create_channel(): line 781] zerocopy for eth6 --> eth6{0 (index: 0) enabled > > [netmap_create_channel(): line 786] Created netmap:eth6{0 interface > > [netmap_create_channel(): line 746] brick: 0xfac090, local_desc: 0xfac780 > > 972.343600 nm_open [444] overriding ARG3 0 > > [netmap_create_channel(): line 781] zerocopy for eth6 --> eth6{1 (index: 1) enabled > > [netmap_create_channel(): line 786] Created netmap:eth6{1 interface > > [netmap_create_channel(): line 746] brick: 0xfac090, local_desc: 0xfac780 > > 972.344200 nm_open [444] overriding ARG3 0 > > [netmap_create_channel(): line 781] zerocopy for eth6 --> eth6{2 (index: 2) enabled > > [netmap_create_channel(): line 786] Created netmap:eth6{2 interface > > [netmap_create_channel(): line 746] brick: 0xfac090, local_desc: 0xfac780 > > 972.344696 nm_open [444] overriding ARG3 0 > > [netmap_create_channel(): line 781] zerocopy for eth6 --> eth6{3 (index: 3) enabled > > [netmap_create_channel(): line 786] Created netmap:eth6{3 interface > > bricks> pe:start() > > and the related dmesg data is: > > > dmesg: > > ixgbe 0000:81:00.0: eth6: detected SFP+: 5 > > ixgbe 0000:81:00.0: eth6: NIC Link is Up 10 Gbps, Flow Control: RX/TX > > 494.566450 [ 131] ixgbe_netmap_configure_srrctl bufsz: 2048 srrctl: 2 > > ixgbe 0000:81:00.0: eth6: detected SFP+: 5 > > ixgbe 0000:81:00.0: eth6: NIC Link is Up 10 Gbps, Flow Control: RX/TX > > 496.743920 [ 320] netmap_pipe_krings_create ffff880876731a00: case 1, create both ends > > 496.744464 [ 320] netmap_pipe_krings_create ffff880876731000: case 1, create both ends > > 496.745026 [ 320] netmap_pipe_krings_create ffff880878fcb600: case 1, create both ends > > 496.745520 [ 320] netmap_pipe_krings_create ffff880875e06c00: case 1, create both ends > > Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-netmap instead > > Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-netmap instead > > > When I run the "./pkt-gen -i netmap:eth6}0 -f rx" command, I am seeing > only a tiny fraction of the expected traffic: > > > [root at xdev-w4 bin]# ./pkt-gen -i netmap:eth6}0 -f rx > > 007.093750 main [2552] interface is netmap:eth6}0 > > 007.093855 main [2675] running on 1 cpus (have 32) > > 007.094406 extract_ip_range [465] range is 10.0.0.1:1234 to 10.0.0.1:1234 > > 007.094418 extract_ip_range [465] range is 10.1.0.1:1234 to 10.1.0.1:1234 > > 007.094481 main [2770] mapped 334980KB at 0x7f325200d000 > > Receiving from netmap:eth6}0: 1 queues, 1 threads and 1 cpus. > > 007.094525 start_threads [2235] Wait 2 secs for phy reset > > 009.094811 start_threads [2237] Ready... > > 009.094927 receiver_body [1638] reading from netmap:eth6}0 fd 3 main_fd 3 > > 010.095975 main_thread [2325] 3.573 Kpps (3.577 Kpkts 2.151 Mbps in 1001085 usec) 511.00 avg_batch 0 min_space > > 011.097211 main_thread [2325] 2.552 Kpps (2.555 Kpkts 1.643 Mbps in 1001237 usec) 511.00 avg_batch 1 min_space > > 012.098314 main_thread [2325] 3.063 Kpps (3.066 Kpkts 1.981 Mbps in 1001103 usec) 511.00 avg_batch 1 min_space > ... > > ^C021.032505 sigint_h [512] received control-C on thread 0x7f326672f700 > > 021.032531 main_thread [2325] 2.762 Kpps (2.555 Kpkts 1.306 Mbps in 925126 usec) 511.00 avg_batch 1 min_space > > 022.033620 main_thread [2325] 510.000 pps (511.000 pkts 306.248 Kbps in 1001087 usec) 511.00 avg_batch 1 min_space > > Received 33726 packets 2876632 bytes 66 events 85 bytes each in 12.02 seconds. > > Speed: 2.806 Kpps Bandwidth: 1.915 Mbps (raw 2.453 Mbps). Average batch: 511.00 pkts > > Running all 4 netmap interfaces provides about the same volume of data > (which is in total ~ 1% of what I would expect). The cpu usage of the > bricks command is ~ 15-20% regardless of running pkt-gen. > > What can I do differently to get better performance? > > (3) Accessing the interfaces - as far as actually using the interfaces > with bro or tcpdump I am somewhat at a loss. I installed netmap-libpcap > version 1.6.0-PRE-GIT_2016_11_26 and compiled tcpdump: > > [root at xdev-w4 tcpdump-4.6.2]# ./tcpdump --version > tcpdump version 4.6.2 > libpcap version 1.6.0-PRE-GIT_2016_11_26 > OpenSSL 1.0.1e-fips 11 Feb 2013 > > which is the recommended version per the information on the packet > bricks git README. > > I am unable to get either the native or the netmap-libpcap version of > tcpdump to recognize the interfaces: > > [root at xdev-w4 tcpdump-4.6.2]# ./tcpdump -n -i netmap:eth6{0 > tcpdump: netmap:eth6{0: No such device exists > (SIOCGIFHWADDR: No such device) > > The ifconfig info for the interface looks like: > > > eth6 Link encap:Ethernet HWaddr 00:1B:21:9D:95:EA > > UP BROADCAST RUNNING PROMISC MULTICAST MTU:9000 Metric:1 > > RX packets:39463364964 errors:0 dropped:72437 overruns:0 frame:0 > > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:1000 > > RX bytes:56963040769452 (51.8 TiB) TX bytes:0 (0.0 b) > > > At this point I am wondering where to go next. It would be great to use > PB instead of pf_ring, but I will need some help to get there. > > many thanks! > scott > > > > > -- > "The enemy knows the system." > ? Claude Shannon > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From ajamshed at ICSI.Berkeley.EDU Mon Nov 28 22:58:47 2016 From: ajamshed at ICSI.Berkeley.EDU (Asim Jamshed) Date: Tue, 29 Nov 2016 15:58:47 +0900 Subject: [Bro-Dev] Packet Brick question(s) In-Reply-To: <551d1832-bf3a-8847-025e-8f4000363f97@lbl.gov> References: <551d1832-bf3a-8847-025e-8f4000363f97@lbl.gov> Message-ID: Hi Scott, Please see my answers below: On 11/29/2016 01:11 AM, Scott Campbell wrote: > I have been investigating the use of Packet Bricks/netmap as a > replacement for pf_ring on linux, but have a few questions. > > (1) Is there any documentation except for the git page and the scripts > themselves? The script comments are nice and useful, but at times the > syntax is rather opaque. At the moment, there is no other documentation. I will try to add more description to the script to explain each line in further detail. > > (2) Following directions and mailing list recommendations, I have a > working version which reads from a heavily loaded 10G ixgbe interface > and splits the traffic into 4 netmap interfaces. The script looks like: > > utilObj:enable_nmpipes() > pe = PktEngine.new("e0", 1024, 8) > lb = Brick.new("LoadBalancer", 2) > lb:connect_input("eth6") > lb:connect_output("eth6{0", "eth6{1", "eth6{2", "eth6{3") > pe:link(lb) > pe:start() > > where eth6 is the data source interface. > > Script output looks like: >> [root at xdev-w4 PB_INSTALL]# sbin/bricks -f etc/bricks-scripts/startup-one-thread.lua >> [ pmain(): line 466] Executing etc/bricks-scripts/startup-one-thread.lua >> [print_version(): line 348] BRICKS Version 0.5-beta >> bricks> utilObj:enable_nmpipes() >> bricks> pe = PktEngine.new("e0", 1024, 8) >> bricks> lb = Brick.new("LoadBalancer", 2) >> bricks> lb:connect_input("eth6") >> bricks> lb:connect_output("eth6{0", "eth6{1", "eth6{2", "eth6{3") >> bricks> pe:link(lb) >> [ lb_init(): line 66] Adding brick eth6{0 to the engine >> [ promisc(): line 96] Interface eth6 is already set to promiscuous mode >> 970.328612 nm_open [444] overriding ARG3 0 >> 970.328631 nm_open [457] overriding ifname eth6 ringid 0x0 flags 0x1 >> [netmap_link_iface(): line 183] Wait for 2 secs for phy reset >> [brick_link(): line 113] Linking e0 with link eth6 with batch size: 512 and qid: -1 >> [netmap_create_channel(): line 746] brick: 0xfac090, local_desc: 0xfac780 >> 972.343050 nm_open [444] overriding ARG3 0 >> [netmap_create_channel(): line 781] zerocopy for eth6 --> eth6{0 (index: 0) enabled >> [netmap_create_channel(): line 786] Created netmap:eth6{0 interface >> [netmap_create_channel(): line 746] brick: 0xfac090, local_desc: 0xfac780 >> 972.343600 nm_open [444] overriding ARG3 0 >> [netmap_create_channel(): line 781] zerocopy for eth6 --> eth6{1 (index: 1) enabled >> [netmap_create_channel(): line 786] Created netmap:eth6{1 interface >> [netmap_create_channel(): line 746] brick: 0xfac090, local_desc: 0xfac780 >> 972.344200 nm_open [444] overriding ARG3 0 >> [netmap_create_channel(): line 781] zerocopy for eth6 --> eth6{2 (index: 2) enabled >> [netmap_create_channel(): line 786] Created netmap:eth6{2 interface >> [netmap_create_channel(): line 746] brick: 0xfac090, local_desc: 0xfac780 >> 972.344696 nm_open [444] overriding ARG3 0 >> [netmap_create_channel(): line 781] zerocopy for eth6 --> eth6{3 (index: 3) enabled >> [netmap_create_channel(): line 786] Created netmap:eth6{3 interface >> bricks> pe:start() > and the related dmesg data is: > >> dmesg: >> ixgbe 0000:81:00.0: eth6: detected SFP+: 5 >> ixgbe 0000:81:00.0: eth6: NIC Link is Up 10 Gbps, Flow Control: RX/TX >> 494.566450 [ 131] ixgbe_netmap_configure_srrctl bufsz: 2048 srrctl: 2 >> ixgbe 0000:81:00.0: eth6: detected SFP+: 5 >> ixgbe 0000:81:00.0: eth6: NIC Link is Up 10 Gbps, Flow Control: RX/TX >> 496.743920 [ 320] netmap_pipe_krings_create ffff880876731a00: case 1, create both ends >> 496.744464 [ 320] netmap_pipe_krings_create ffff880876731000: case 1, create both ends >> 496.745026 [ 320] netmap_pipe_krings_create ffff880878fcb600: case 1, create both ends >> 496.745520 [ 320] netmap_pipe_krings_create ffff880875e06c00: case 1, create both ends >> Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-netmap instead >> Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-netmap instead > > When I run the "./pkt-gen -i netmap:eth6}0 -f rx" command, I am seeing > only a tiny fraction of the expected traffic: > >> [root at xdev-w4 bin]# ./pkt-gen -i netmap:eth6}0 -f rx >> 007.093750 main [2552] interface is netmap:eth6}0 >> 007.093855 main [2675] running on 1 cpus (have 32) >> 007.094406 extract_ip_range [465] range is 10.0.0.1:1234 to 10.0.0.1:1234 >> 007.094418 extract_ip_range [465] range is 10.1.0.1:1234 to 10.1.0.1:1234 >> 007.094481 main [2770] mapped 334980KB at 0x7f325200d000 >> Receiving from netmap:eth6}0: 1 queues, 1 threads and 1 cpus. >> 007.094525 start_threads [2235] Wait 2 secs for phy reset >> 009.094811 start_threads [2237] Ready... >> 009.094927 receiver_body [1638] reading from netmap:eth6}0 fd 3 main_fd 3 >> 010.095975 main_thread [2325] 3.573 Kpps (3.577 Kpkts 2.151 Mbps in 1001085 usec) 511.00 avg_batch 0 min_space >> 011.097211 main_thread [2325] 2.552 Kpps (2.555 Kpkts 1.643 Mbps in 1001237 usec) 511.00 avg_batch 1 min_space >> 012.098314 main_thread [2325] 3.063 Kpps (3.066 Kpkts 1.981 Mbps in 1001103 usec) 511.00 avg_batch 1 min_space > ... >> ^C021.032505 sigint_h [512] received control-C on thread 0x7f326672f700 >> 021.032531 main_thread [2325] 2.762 Kpps (2.555 Kpkts 1.306 Mbps in 925126 usec) 511.00 avg_batch 1 min_space >> 022.033620 main_thread [2325] 510.000 pps (511.000 pkts 306.248 Kbps in 1001087 usec) 511.00 avg_batch 1 min_space >> Received 33726 packets 2876632 bytes 66 events 85 bytes each in 12.02 seconds. >> Speed: 2.806 Kpps Bandwidth: 1.915 Mbps (raw 2.453 Mbps). Average batch: 511.00 pkts > Running all 4 netmap interfaces provides about the same volume of data > (which is in total ~ 1% of what I would expect). The cpu usage of the > bricks command is ~ 15-20% regardless of running pkt-gen. > > What can I do differently to get better performance? I will first like to know a little bit more about your setup. I tested packet-brickson linux-3.13.0 kernel (I know it's old.. sorry). Can you please share with me thekernel version as well as the netmap driver version? I want to recreateyour setup in my testbed and see what the problem is. Also, please tryto see how much traffic (bandwidth/pps) you can get without using netmap pipes, i.e.: ./pkt-gen -i netmap:eth6 -f rx I will like to know whether your netmap driver settings are okay. > > (3) Accessing the interfaces - as far as actually using the interfaces > with bro or tcpdump I am somewhat at a loss. I installed netmap-libpcap > version 1.6.0-PRE-GIT_2016_11_26 and compiled tcpdump: > > [root at xdev-w4 tcpdump-4.6.2]# ./tcpdump --version > tcpdump version 4.6.2 > libpcap version 1.6.0-PRE-GIT_2016_11_26 > OpenSSL 1.0.1e-fips 11 Feb 2013 > > which is the recommended version per the information on the packet > bricks git README. > > I am unable to get either the native or the netmap-libpcap version of > tcpdump to recognize the interfaces: > > [root at xdev-w4 tcpdump-4.6.2]# ./tcpdump -n -i netmap:eth6{0 > tcpdump: netmap:eth6{0: No such device exists > (SIOCGIFHWADDR: No such device) Were you running packet-bricks when you started tcpdump? Please also try running the following command and see if tcpdump works: ./tcpdump -n -i netmap:eth6 > The ifconfig info for the interface looks like: > >> eth6 Link encap:Ethernet HWaddr 00:1B:21:9D:95:EA >> UP BROADCAST RUNNING PROMISC MULTICAST MTU:9000 Metric:1 >> RX packets:39463364964 errors:0 dropped:72437 overruns:0 frame:0 >> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:1000 >> RX bytes:56963040769452 (51.8 TiB) TX bytes:0 (0.0 b) > > At this point I am wondering where to go next. It would be great to use > PB instead of pf_ring, but I will need some help to get there. > > many thanks! > scott Regards, --Asim > > > From asharma at lbl.gov Tue Nov 29 09:06:10 2016 From: asharma at lbl.gov (Aashish Sharma) Date: Tue, 29 Nov 2016 09:06:10 -0800 Subject: [Bro-Dev] bro-pkg upgrade and over-writing of files Message-ID: <20161129170609.GO67232@mac-799.local> Hello, I have a package where I provide a sample configuration file for people to redef according to their needs and specifics. Now everytime when they upgrade the package, I risk over writing their modified config file. SO I decided to call the config file scan-config.bro.orig but then I am running into issues of which one to load and how to determine the presence of an already existing scan-config.bro in __load__.bro The idea of asking uses to redef outside package directory might be cumbersome for unfamiliar users. Any thoughts ? Aashish From jsiwek at illinois.edu Tue Nov 29 09:42:39 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Tue, 29 Nov 2016 17:42:39 +0000 Subject: [Bro-Dev] Package manager meta data In-Reply-To: <63694BF3-3E9D-4419-B31D-B96C9E980364@illinois.edu> References: <20161027232819.GD20933@icir.org> <9A7795D3-3C47-475B-AEB1-2CEDF9BB7AE4@illinois.edu> <12c859cf-b998-e726-c852-1caa32333e16@gmail.com> <536BD67A-7A29-4B3F-9ABB-38DA079C53F3@illinois.edu> <1760cb97-de75-1cf7-b42e-899e4688418f@gmail.com> <1303E27E-FC8D-4FA5-AD68-DA64FC401F84@illinois.edu> <63694BF3-3E9D-4419-B31D-B96C9E980364@illinois.edu> Message-ID: <8A9CBA4C-4527-49A1-8F38-C57FC79D6F74@illinois.edu> The metadata changes discussed in this thread are now in bro-pkg 0.8 (and the structure of the bro/packages source on GitHub is adapted for it). - Jon From jsiwek at illinois.edu Tue Nov 29 11:51:21 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Tue, 29 Nov 2016 19:51:21 +0000 Subject: [Bro-Dev] bro-pkg upgrade and over-writing of files In-Reply-To: <20161129170609.GO67232@mac-799.local> References: <20161129170609.GO67232@mac-799.local> Message-ID: <5C96343F-76A6-4A1F-B365-AE494F6C8E82@illinois.edu> > On Nov 29, 2016, at 11:06 AM, Aashish Sharma wrote: > > I have a package where I provide a sample configuration file for people to redef according to their needs and specifics. > > Now everytime when they upgrade the package, I risk over writing their modified config file. > > SO I decided to call the config file scan-config.bro.orig but then I am running into issues of which one to load and how to determine the presence of an already existing scan-config.bro in __load__.bro > > The idea of asking uses to redef outside package directory might be cumbersome for unfamiliar users. > > Any thoughts ? At the moment, the best thing would be to ask users to explicitly copy/modify/@load the sample config file somewhere in the site/ dir. But a new feature could be added to bro-pkg that allows package authors to specify a list of config files in their bro-pkg.meta. Then on install/upgrade/remove, if a user has made modifications to any of those files, they can be warned/prompted about how to proceed (show a diff, ask to overwrite or keep modified version, etc.). This seems a common way to handle config files in the package management scene. Would such a feature work for you? - Jon From asharma at lbl.gov Tue Nov 29 11:58:54 2016 From: asharma at lbl.gov (Aashish Sharma) Date: Tue, 29 Nov 2016 11:58:54 -0800 Subject: [Bro-Dev] bro-pkg upgrade and over-writing of files In-Reply-To: <5C96343F-76A6-4A1F-B365-AE494F6C8E82@illinois.edu> References: <20161129170609.GO67232@mac-799.local> <5C96343F-76A6-4A1F-B365-AE494F6C8E82@illinois.edu> Message-ID: <20161129195853.GB3305@mac-799.local> On Tue, Nov 29, 2016 at 07:51:21PM +0000, Siwek, Jon wrote: > > But a new feature could be added to bro-pkg that allows package authors to specify a list of config files in their bro-pkg.meta. Then on install/upgrade/remove, if a user has made modifications to any of those files, they can be warned/prompted about how to proceed (show a diff, ask to overwrite or keep modified version, etc.). This seems a common way to handle config files in the package management scene. > I like the idea of warned/prompted (show a diff, ask to overwrite or keep new version etc) is good. That helps with situations where a new variables introduced in in a config file after the upgrade also don't go unnoticed. > Would such a feature work for you? Yes absolutely. This would be rather perfect. Thank you, Aashish