[Bro] Bro and APCON

Josh Guild josh.guild at morphick.com
Wed Apr 27 07:14:41 PDT 2016


Hey Dave,

Still doing some preliminary analysis but we had one of our clients strip
the VLAN protocol on our egress ports from the APCON and it allowed
Bro/PF_Ring to sessionize the traffic properly (we had tried different
-tuples in our set up previously). We have a test we run that checks each
level of our framework to make sure we have the proper visibility. This was
always a scattershot of what was logged by Bro but, after we stripped the
VLAN protocol going to our box, it was cleaned up and looked good.

I'm going to have them strip the FabricPath protocol and see how that
affects the traffic as well (can only strip one protocol at a time).

Odd thing is, the weird.log entries were still roughly the same with the
VLANs stripped, so it was something with how Bro or PF_Ring was handling
the incoming packets from the APCON.

Research continues!

On Thu, Apr 21, 2016 at 8:07 PM, Josh Guild <josh.guild at morphick.com> wrote:

> That's awesome that you got it worked out! I'll take a look and see what
> the configs are for the APCONs we're dealing with and maybe get to the
> bottom of our issue.
>
> Thanks!
>
> On Thu, Apr 21, 2016, 19:58 Dave Crawford <bro at pingtrip.com> wrote:
>
>> Just providing a quick update that think we solved our issue. Our SPANs
>> began getting FabricPath packets after the network team upgraded to Nexus
>> switches (see my separate thread about workers at 100% CPU).
>>
>> Long story short, Apcon has a "Protocol Stripping" configuration screen
>> where you can enable stripping of the FP encapsulation layer. After
>> enabling that option the "data_before_established" and "inappropriate_FIN"
>> messages stopped.
>>
>> -Dave
>>
>>
>> On Apr 21, 2016, at 12:22 PM, Josh Guild <josh.guild at morphick.com> wrote:
>>
>> Yep, in one of the environments, we're getting a ton of
>> "possible_split_routing" and "data_before_established" both with and
>> without the APCON in the mix.
>> Thinking it has to do with how Bro is handling the load balancing to
>> workers.
>>
>> On Thu, Apr 21, 2016 at 10:45 AM, Dave Crawford <bro at pingtrip.com> wrote:
>>
>>> Thanks Josh,
>>>
>>> We’re also trying to determine if the Apcon is a red herring since
>>> (unfortunately) two changes were made at the same time. While we swapped
>>> our Anues for Apcons the network team was also upgrading to Nexus switches.
>>>
>>> Our weird log started filling with with “data_before_established” and
>>> “possible_split_routing” events right after the changes.
>>>
>>> -Dave
>>>
>>>
>>> On Apr 21, 2016, at 10:30 AM, Josh Guild <josh.guild at morphick.com>
>>> wrote:
>>>
>>> Hi Dave,
>>>
>>> We bypassed the APCON in one of the environments and it helped a little
>>> with capture loss (about a 10% drop) and errors in the weird.log.
>>> Unfortunately, this was during a weekend so it's tough to say how much of
>>> an impact it made. Another network we're in fixed some load balancing
>>> issues upstream and this help significantly with loss (though weird.logs
>>> remain about where they were). I think the APCON may have been a red
>>> herring in this instance but I'd be curious to see how your network looks
>>> before and after implementation if you'd like to keep in touch.
>>>
>>> The main things I've been looking at are capture loss and weird.log
>>> errors (specifically HTTP_version_mismatch, SYN_seq_jump,
>>> TCP_seq_underflow_or_misorder) these may lean towards traffic being
>>> mangled. This presentation is pretty helpful in showing you what to look
>>> for:
>>> https://speakerdeck.com/vladg/bro-deployment-verification-and-troubleshooting
>>>
>>> Thanks!
>>>
>>> On Thu, Apr 21, 2016 at 10:08 AM, Dave Crawford <dave at pingtrip.com>
>>> wrote:
>>>
>>>> Josh,
>>>>
>>>> Were you able to solve this issue? We just started swapping out our
>>>> current solution with Apcon’s and wondering if we’ll run into the same
>>>> issue.
>>>>
>>>> -Dave
>>>>
>>>> On Apr 7, 2016, at 2:39 PM, Josh Guild <josh.guild at morphick.com> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> We have a few deployments that utilize an APCON for traffic
>>>> aggregation. We've noticed in these environments that Bro has trouble
>>>> reassembling the traffic correctly and there is a significant amount of
>>>> capture loss (based on the script). We've tried different hashing
>>>> algorithms on the APCON to no effect.
>>>>
>>>> Has anyone else seen anything similar to this or have any insight?
>>>>
>>>> Thanks!
>>>>
>>>> --
>>>> Josh Guild
>>>> Network Intelligence Analyst
>>>> <https://twitter.com/stay_spooky> <https://keybase.io/joshuaguild>
>>>>
>>>> _______________________________________________
>>>> Bro mailing list
>>>> bro at bro-ids.org
>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>> <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Josh Guild
>>> Network Intelligence Analyst
>>> <https://twitter.com/stay_spooky> <https://keybase.io/joshuaguild>
>>>
>>>
>>>
>>
>>
>> --
>> Josh Guild
>> Network Intelligence Analyst
>> <https://twitter.com/stay_spooky> <https://keybase.io/joshuaguild>
>>
>>
>>


-- 
Josh Guild
Network Intelligence Analyst
<https://twitter.com/stay_spooky> <https://keybase.io/joshuaguild>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160427/291c89fd/attachment-0001.html 


More information about the Bro mailing list