[Bro] So uh...how do you know which pin_cpus to use?

[Michał Purzyński]

Lesson learned for me. Never answer from a phone, esp. trying to cover numa
allocation on 56 threads on 4 inches ;)

Take back what I said. Here is how it looks like, I'm in front of a server
with 2x NIC. I have E5-2697 v3 here, 14 physical cores per CPU, HT enabled,
kernel 4.4.something.

0-13 - NUMA node 0, CPU 0, hthreads 0-13
14-27 - NUMA node 1, CPU 1, cores 14-27
28-41 - NUMA node 0, CPU 0, hthreads 28-41
42-55 - NUMA node 1, CPU 1 again

1st card should use virtual cores (AKA threads) 0-13 + 28-41
2nd card should use 14-27 + 42-55



On Wed, Oct 19, 2016 at 12:28 AM, Azoff, Justin S <jazoff at illinois.edu>
wrote:

>
> > On Oct 18, 2016, at 6:18 PM, Michał Purzyński <
> michalpurzynski1 at gmail.com> wrote:
> >
> > 2.6 kernels on Linux enumerate HT in a different way 3.x and 4.x do
> >
> > 2.6
> >
> > Core 0 thread 0
> > Core 0 thread 1
> >
> > Etc
> >
> > 3.x
> >
> > Core 0-N on CPU 0 first half of threads
> > Then CPU 1
> > Then CPU 0 second half of threads
> > Then CPU 1
> >
> > Results for HT vs cross numa are about to be published, soon ;)
> > I don't like cache misses when CPU 1 is reaching for data on node 0
> though. It is not about cross numa bandwidth it's the fact then you have in
> the worst case 67ns to process a smallest packet on 10Gbit. And L3 hit on
> ivy bridge is at least 15ns.
> > Miss is 5x that.
>
> Ah!  That explains a lot.  I wonder if numa allocation changed too.  We
> just upgraded some machines from centos6 to 7 and I was wondering how the
> meticulously written node.cfg we had been using for months now appeared
> completely wrong.
>
> I wonder if broctl should support hwloc for cpu pinning instead of task
> set.  I wouldn't mind having an 'auto' mode that just does the right thing.
>
> It looks like on our dual socket numa box we should be using
>
> 0,2,4,6,8,10,12,14 for one 10g card and
> 1,3,5,7,9,11,13,15 for the other 10g card
>
> 0-19 are the physical cores and 20-39 are the HT cores, but using 0,1,2,3
> flips between numa nodes which is not what anyone wants.
>
>
> --
> - Justin Azoff
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161019/d1b4f9b7/attachment.html