[Bro-Dev] Packet Brick question(s)
Scott Campbell
scampbell at lbl.gov
Mon Nov 28 08:11:08 PST 2016
I have been investigating the use of Packet Bricks/netmap as a
replacement for pf_ring on linux, but have a few questions.
(1) Is there any documentation except for the git page and the scripts
themselves? The script comments are nice and useful, but at times the
syntax is rather opaque.
(2) Following directions and mailing list recommendations, I have a
working version which reads from a heavily loaded 10G ixgbe interface
and splits the traffic into 4 netmap interfaces. The script looks like:
utilObj:enable_nmpipes()
pe = PktEngine.new("e0", 1024, 8)
lb = Brick.new("LoadBalancer", 2)
lb:connect_input("eth6")
lb:connect_output("eth6{0", "eth6{1", "eth6{2", "eth6{3")
pe:link(lb)
pe:start()
where eth6 is the data source interface.
Script output looks like:
> [root at xdev-w4 PB_INSTALL]# sbin/bricks -f etc/bricks-scripts/startup-one-thread.lua
> [ pmain(): line 466] Executing etc/bricks-scripts/startup-one-thread.lua
> [print_version(): line 348] BRICKS Version 0.5-beta
> bricks> utilObj:enable_nmpipes()
> bricks> pe = PktEngine.new("e0", 1024, 8)
> bricks> lb = Brick.new("LoadBalancer", 2)
> bricks> lb:connect_input("eth6")
> bricks> lb:connect_output("eth6{0", "eth6{1", "eth6{2", "eth6{3")
> bricks> pe:link(lb)
> [ lb_init(): line 66] Adding brick eth6{0 to the engine
> [ promisc(): line 96] Interface eth6 is already set to promiscuous mode
> 970.328612 nm_open [444] overriding ARG3 0
> 970.328631 nm_open [457] overriding ifname eth6 ringid 0x0 flags 0x1
> [netmap_link_iface(): line 183] Wait for 2 secs for phy reset
> [brick_link(): line 113] Linking e0 with link eth6 with batch size: 512 and qid: -1
> [netmap_create_channel(): line 746] brick: 0xfac090, local_desc: 0xfac780
> 972.343050 nm_open [444] overriding ARG3 0
> [netmap_create_channel(): line 781] zerocopy for eth6 --> eth6{0 (index: 0) enabled
> [netmap_create_channel(): line 786] Created netmap:eth6{0 interface
> [netmap_create_channel(): line 746] brick: 0xfac090, local_desc: 0xfac780
> 972.343600 nm_open [444] overriding ARG3 0
> [netmap_create_channel(): line 781] zerocopy for eth6 --> eth6{1 (index: 1) enabled
> [netmap_create_channel(): line 786] Created netmap:eth6{1 interface
> [netmap_create_channel(): line 746] brick: 0xfac090, local_desc: 0xfac780
> 972.344200 nm_open [444] overriding ARG3 0
> [netmap_create_channel(): line 781] zerocopy for eth6 --> eth6{2 (index: 2) enabled
> [netmap_create_channel(): line 786] Created netmap:eth6{2 interface
> [netmap_create_channel(): line 746] brick: 0xfac090, local_desc: 0xfac780
> 972.344696 nm_open [444] overriding ARG3 0
> [netmap_create_channel(): line 781] zerocopy for eth6 --> eth6{3 (index: 3) enabled
> [netmap_create_channel(): line 786] Created netmap:eth6{3 interface
> bricks> pe:start()
and the related dmesg data is:
> dmesg:
> ixgbe 0000:81:00.0: eth6: detected SFP+: 5
> ixgbe 0000:81:00.0: eth6: NIC Link is Up 10 Gbps, Flow Control: RX/TX
> 494.566450 [ 131] ixgbe_netmap_configure_srrctl bufsz: 2048 srrctl: 2
> ixgbe 0000:81:00.0: eth6: detected SFP+: 5
> ixgbe 0000:81:00.0: eth6: NIC Link is Up 10 Gbps, Flow Control: RX/TX
> 496.743920 [ 320] netmap_pipe_krings_create ffff880876731a00: case 1, create both ends
> 496.744464 [ 320] netmap_pipe_krings_create ffff880876731000: case 1, create both ends
> 496.745026 [ 320] netmap_pipe_krings_create ffff880878fcb600: case 1, create both ends
> 496.745520 [ 320] netmap_pipe_krings_create ffff880875e06c00: case 1, create both ends
> Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-netmap instead
> Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-netmap instead
When I run the "./pkt-gen -i netmap:eth6}0 -f rx" command, I am seeing
only a tiny fraction of the expected traffic:
> [root at xdev-w4 bin]# ./pkt-gen -i netmap:eth6}0 -f rx
> 007.093750 main [2552] interface is netmap:eth6}0
> 007.093855 main [2675] running on 1 cpus (have 32)
> 007.094406 extract_ip_range [465] range is 10.0.0.1:1234 to 10.0.0.1:1234
> 007.094418 extract_ip_range [465] range is 10.1.0.1:1234 to 10.1.0.1:1234
> 007.094481 main [2770] mapped 334980KB at 0x7f325200d000
> Receiving from netmap:eth6}0: 1 queues, 1 threads and 1 cpus.
> 007.094525 start_threads [2235] Wait 2 secs for phy reset
> 009.094811 start_threads [2237] Ready...
> 009.094927 receiver_body [1638] reading from netmap:eth6}0 fd 3 main_fd 3
> 010.095975 main_thread [2325] 3.573 Kpps (3.577 Kpkts 2.151 Mbps in 1001085 usec) 511.00 avg_batch 0 min_space
> 011.097211 main_thread [2325] 2.552 Kpps (2.555 Kpkts 1.643 Mbps in 1001237 usec) 511.00 avg_batch 1 min_space
> 012.098314 main_thread [2325] 3.063 Kpps (3.066 Kpkts 1.981 Mbps in 1001103 usec) 511.00 avg_batch 1 min_space
...
> ^C021.032505 sigint_h [512] received control-C on thread 0x7f326672f700
> 021.032531 main_thread [2325] 2.762 Kpps (2.555 Kpkts 1.306 Mbps in 925126 usec) 511.00 avg_batch 1 min_space
> 022.033620 main_thread [2325] 510.000 pps (511.000 pkts 306.248 Kbps in 1001087 usec) 511.00 avg_batch 1 min_space
> Received 33726 packets 2876632 bytes 66 events 85 bytes each in 12.02 seconds.
> Speed: 2.806 Kpps Bandwidth: 1.915 Mbps (raw 2.453 Mbps). Average batch: 511.00 pkts
Running all 4 netmap interfaces provides about the same volume of data
(which is in total ~ 1% of what I would expect). The cpu usage of the
bricks command is ~ 15-20% regardless of running pkt-gen.
What can I do differently to get better performance?
(3) Accessing the interfaces - as far as actually using the interfaces
with bro or tcpdump I am somewhat at a loss. I installed netmap-libpcap
version 1.6.0-PRE-GIT_2016_11_26 and compiled tcpdump:
[root at xdev-w4 tcpdump-4.6.2]# ./tcpdump --version
tcpdump version 4.6.2
libpcap version 1.6.0-PRE-GIT_2016_11_26
OpenSSL 1.0.1e-fips 11 Feb 2013
which is the recommended version per the information on the packet
bricks git README.
I am unable to get either the native or the netmap-libpcap version of
tcpdump to recognize the interfaces:
[root at xdev-w4 tcpdump-4.6.2]# ./tcpdump -n -i netmap:eth6{0
tcpdump: netmap:eth6{0: No such device exists
(SIOCGIFHWADDR: No such device)
The ifconfig info for the interface looks like:
> eth6 Link encap:Ethernet HWaddr 00:1B:21:9D:95:EA
> UP BROADCAST RUNNING PROMISC MULTICAST MTU:9000 Metric:1
> RX packets:39463364964 errors:0 dropped:72437 overruns:0 frame:0
> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:56963040769452 (51.8 TiB) TX bytes:0 (0.0 b)
At this point I am wondering where to go next. It would be great to use
PB instead of pf_ring, but I will need some help to get there.
many thanks!
scott
--
"The enemy knows the system."
— Claude Shannon
More information about the bro-dev
mailing list