[Bro] So uh...how do you know which pin_cpus to use?
michalpurzynski1 at gmail.com
Tue Oct 18 15:37:44 PDT 2016
Lesson learned for me. Never answer from a phone, esp. trying to cover numa
allocation on 56 threads on 4 inches ;)
Take back what I said. Here is how it looks like, I'm in front of a server
with 2x NIC. I have E5-2697 v3 here, 14 physical cores per CPU, HT enabled,
0-13 - NUMA node 0, CPU 0, hthreads 0-13
14-27 - NUMA node 1, CPU 1, cores 14-27
28-41 - NUMA node 0, CPU 0, hthreads 28-41
42-55 - NUMA node 1, CPU 1 again
1st card should use virtual cores (AKA threads) 0-13 + 28-41
2nd card should use 14-27 + 42-55
On Wed, Oct 19, 2016 at 12:28 AM, Azoff, Justin S <jazoff at illinois.edu>
> > On Oct 18, 2016, at 6:18 PM, Michał Purzyński <
> michalpurzynski1 at gmail.com> wrote:
> > 2.6 kernels on Linux enumerate HT in a different way 3.x and 4.x do
> > 2.6
> > Core 0 thread 0
> > Core 0 thread 1
> > Etc
> > 3.x
> > Core 0-N on CPU 0 first half of threads
> > Then CPU 1
> > Then CPU 0 second half of threads
> > Then CPU 1
> > Results for HT vs cross numa are about to be published, soon ;)
> > I don't like cache misses when CPU 1 is reaching for data on node 0
> though. It is not about cross numa bandwidth it's the fact then you have in
> the worst case 67ns to process a smallest packet on 10Gbit. And L3 hit on
> ivy bridge is at least 15ns.
> > Miss is 5x that.
> Ah! That explains a lot. I wonder if numa allocation changed too. We
> just upgraded some machines from centos6 to 7 and I was wondering how the
> meticulously written node.cfg we had been using for months now appeared
> completely wrong.
> I wonder if broctl should support hwloc for cpu pinning instead of task
> set. I wouldn't mind having an 'auto' mode that just does the right thing.
> It looks like on our dual socket numa box we should be using
> 0,2,4,6,8,10,12,14 for one 10g card and
> 1,3,5,7,9,11,13,15 for the other 10g card
> 0-19 are the physical cores and 20-39 are the HT cores, but using 0,1,2,3
> flips between numa nodes which is not what anyone wants.
> - Justin Azoff
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Bro