[Bro] Bro and APCON

Josh Guild josh.guild at morphick.com
Thu Apr 21 17:07:23 PDT 2016


That's awesome that you got it worked out! I'll take a look and see what
the configs are for the APCONs we're dealing with and maybe get to the
bottom of our issue.

Thanks!

On Thu, Apr 21, 2016, 19:58 Dave Crawford <bro at pingtrip.com> wrote:

> Just providing a quick update that think we solved our issue. Our SPANs
> began getting FabricPath packets after the network team upgraded to Nexus
> switches (see my separate thread about workers at 100% CPU).
>
> Long story short, Apcon has a "Protocol Stripping" configuration screen
> where you can enable stripping of the FP encapsulation layer. After
> enabling that option the "data_before_established" and "inappropriate_FIN"
> messages stopped.
>
> -Dave
>
>
> On Apr 21, 2016, at 12:22 PM, Josh Guild <josh.guild at morphick.com> wrote:
>
> Yep, in one of the environments, we're getting a ton of
> "possible_split_routing" and "data_before_established" both with and
> without the APCON in the mix.
> Thinking it has to do with how Bro is handling the load balancing to
> workers.
>
> On Thu, Apr 21, 2016 at 10:45 AM, Dave Crawford <bro at pingtrip.com> wrote:
>
>> Thanks Josh,
>>
>> We’re also trying to determine if the Apcon is a red herring since
>> (unfortunately) two changes were made at the same time. While we swapped
>> our Anues for Apcons the network team was also upgrading to Nexus switches.
>>
>> Our weird log started filling with with “data_before_established” and
>> “possible_split_routing” events right after the changes.
>>
>> -Dave
>>
>>
>> On Apr 21, 2016, at 10:30 AM, Josh Guild <josh.guild at morphick.com> wrote:
>>
>> Hi Dave,
>>
>> We bypassed the APCON in one of the environments and it helped a little
>> with capture loss (about a 10% drop) and errors in the weird.log.
>> Unfortunately, this was during a weekend so it's tough to say how much of
>> an impact it made. Another network we're in fixed some load balancing
>> issues upstream and this help significantly with loss (though weird.logs
>> remain about where they were). I think the APCON may have been a red
>> herring in this instance but I'd be curious to see how your network looks
>> before and after implementation if you'd like to keep in touch.
>>
>> The main things I've been looking at are capture loss and weird.log
>> errors (specifically HTTP_version_mismatch, SYN_seq_jump,
>> TCP_seq_underflow_or_misorder) these may lean towards traffic being
>> mangled. This presentation is pretty helpful in showing you what to look
>> for:
>> https://speakerdeck.com/vladg/bro-deployment-verification-and-troubleshooting
>>
>> Thanks!
>>
>> On Thu, Apr 21, 2016 at 10:08 AM, Dave Crawford <dave at pingtrip.com>
>> wrote:
>>
>>> Josh,
>>>
>>> Were you able to solve this issue? We just started swapping out our
>>> current solution with Apcon’s and wondering if we’ll run into the same
>>> issue.
>>>
>>> -Dave
>>>
>>> On Apr 7, 2016, at 2:39 PM, Josh Guild <josh.guild at morphick.com> wrote:
>>>
>>> Hi all,
>>>
>>> We have a few deployments that utilize an APCON for traffic aggregation.
>>> We've noticed in these environments that Bro has trouble reassembling the
>>> traffic correctly and there is a significant amount of capture loss (based
>>> on the script). We've tried different hashing algorithms on the APCON to no
>>> effect.
>>>
>>> Has anyone else seen anything similar to this or have any insight?
>>>
>>> Thanks!
>>>
>>> --
>>> Josh Guild
>>> Network Intelligence Analyst
>>> <https://twitter.com/stay_spooky> <https://keybase.io/joshuaguild>
>>>
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>> <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
>>>
>>>
>>>
>>
>>
>> --
>> Josh Guild
>> Network Intelligence Analyst
>> <https://twitter.com/stay_spooky> <https://keybase.io/joshuaguild>
>>
>>
>>
>
>
> --
> Josh Guild
> Network Intelligence Analyst
> <https://twitter.com/stay_spooky> <https://keybase.io/joshuaguild>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160422/6a56572b/attachment-0001.html 


More information about the Bro mailing list