[Bro] Troubleshooting crashes

[Tritium Cat]

On Wed, Sep 19, 2012 at 7:19 PM, Seth Hall <seth at icir.org> wrote:

>
> Ok, I found the problem.  We got tripped up when they added the
> configuration option and we may have to do a 2.1.1 release.  We were
> setting the PCAP_PF_RING_USE_CLUSTER_PER_FLOW env var to "1" just to have
> it set to something.  They started using that value to set how the load
> balancing is done and "1" was chosen to be round-robin.
>
> We probably need to make that configurable (from broctl.cfg) with a
> default of 2 (2-tuple… it causes less trouble).
>
> Here's the structure that defines the different load balancing approaches
> with that variable…
>
> typedef enum {
>   cluster_per_flow = 0,     /* 6-tuple: <src ip, src port, dst ip, dst
> port, proto, vlan>  */
>   cluster_round_robin,
>   cluster_per_flow_2_tuple, /* 2-tuple: <src ip,           dst ip
>               >  */
>   cluster_per_flow_4_tuple, /* 4-tuple: <src ip, src port, dst ip, dst
> port             >  */
>   cluster_per_flow_5_tuple, /* 5-tuple: <src ip, src port, dst ip, dst
> port, proto      >  */
> } cluster_type;
>


Ok that's good to hear, however I still think the problem is with PF_RING.
 I'm not sure your assessment of that environment variable is correct.  See
the last three emails to the pf_ring developers here: [1].

>From pf_ring libpcap....

pcap-linux.c.orig:1183:
if(getenv("PCAP_PF_RING_USE_CLUSTER_PER_FLOW"))
pcap-linux.c.orig-1184-           pfring_set_cluster(handle->ring,
atoi(clusterId), cluster_per_flow);
pcap-linux.c.orig-1185-         else
pcap-linux.c.orig-1186-           pfring_set_cluster(handle->ring,
atoi(clusterId), cluster_round_robin);


Notice the call to pfring_set_cluster.

"cluster_per_flow" is the default 5-tuple hashing mode.  (Actually 6-tuple
due to the vlan problem).  Below I've included the relevant code from
pf_ring that shows the only difference between cluster_per_flow
and cluster_per_flow_5_tuple is the call to hash_pkt_header().

"hash_pkt_header" is called from "hash_pkt_cluster" and
"default_rehash_rss_func".


Inside kernel/pf_ring.c, the function "hash_pkt_cluster" will determine the
hashing mode and call another function "hash_pkt_header".  The last
argument to "hash_pkt_header" instructs PF_RING to mask/prevent the VLAN
from being used in the hash calculation.

It looks like this --> idx = hash_pkt_header(hdr, 0, 0, 0, 0, 1);

Notice every call to hash_pkt_header() except the default case of
"cluster_per_flow" attempts to mask the vlan.  (Up until a few days ago,
all calls to hash_pkt_header() did not mask the vlan).


I patched PF_RING in various places to force hash_pkt_header() to always
mask the vlan.. regardless of what environment variable was set.  This did
not work so I took it a step further.


Of course it's always possible I'm highly confused, time will tell :o

/tc



3889:static u_int hash_pkt_cluster(ring_cluster_element *cluster_ptr,
3890-                         struct pfring_pkthdr *hdr)
3891-{
3892-  u_int idx;
3893-
3894-  switch(cluster_ptr->cluster.hashing_mode) {
3895-    case cluster_round_robin:
3896-      idx = cluster_ptr->cluster.hashing_id++;
3897-      break;
3898-
3899-    case cluster_per_flow_2_tuple:
3900-      idx = hash_pkt_header(hdr, 0, 0, 1, 1, 1);
3901-      break;
3902-
3903-    case cluster_per_flow_4_tuple:
3904-      idx = hash_pkt_header(hdr, 0, 0, 0, 1, 1);
3905-      break;
3906-
3907-    case cluster_per_flow_tcp_5_tuple:
3908-      if(((hdr->extended_hdr.parsed_pkt.tunnel.tunnel_id ==
NO_TUNNEL_ID) ?
3909-     hdr->extended_hdr.parsed_pkt.l3_proto :
hdr->extended_hdr.parsed_pkt.tunnel.tunneled_proto) == IPPROTO_TCP)
3910-   idx = hash_pkt_header(hdr, 0, 0, 0, 0, 1); /* 5 tuple */
3911-      else
3912-   idx = hash_pkt_header(hdr, 0, 0, 1, 1, 1);   /* 2 tuple */
3913-      break;
3914-
3915-    case cluster_per_flow_5_tuple:
3916-      idx = hash_pkt_header(hdr, 0, 0, 0, 0, 1);
3917-      break;
3918-
3919-    case cluster_per_flow:
3920-    default:
3921-      idx = hash_pkt_header(hdr, 0, 0, 0, 0, 0);
3922-      break;
3923-  }
3924-
3925-  return(idx % cluster_ptr->cluster.num_cluster_elements);
3926-}
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120919/cc80ccb0/attachment.html