[Bro] So uh...how do you know which pin_cpus to use?

[James Lay]

On 2016-10-18 14:55, Azoff, Justin S wrote:
>> On Oct 18, 2016, at 4:52 PM, Azoff, Justin S <jazoff at illinois.edu> 
>> wrote:
>> 
>> 
>>> On Oct 18, 2016, at 4:34 PM, James Lay <jlay at slave-tothe-box.net> 
>>> wrote:
>>> 
>>> Never really understood this:
>>> 
>>> "The correct pin_cpus setting to use is dependent on your CPU
>>> architecture. Intel and AMD systems enumerate processors in different
>>> ways. Using the wrong pin_cpus setting can cause poor performance."
>>> 
>>> Is there a magical formula?  Any advice would help thanks.
>> 
>> The best thing to do is to install the hwloc package and use the 
>> lstopo or lstopo-no-graphics tool to render a big ascii art image of 
>> the system.
>> 
>> 
>> on centos7 this works:
>> 
>> lstopo-no-graphics  --of txt
>> 
>> 
>> You'll get something that looks like this:
>> 
>> https://www.open-mpi.org/projects/hwloc/lstopo/images/2XeonE5v2+2cuda+1display_v1.11.png
>> 
>> or
>> 
>> https://www.open-mpi.org/projects/hwloc/lstopo/images/4Opteron6200.v1.11.png
>> 
>> The numbers towards the bottom are the cpu ids.  So you can see that 
>> using something like
>> 
>> 1,3,5,7,9,11,13,15,17,19,21,23,25
>> 
>> on an intel cpu would be the worst thing you could do since 21,23,25 
>> are on the same physical cores as 1,3, and 5
> 
> Oh, I should add... ".. on that particular system".  On some of our
> numa machines the allocation is different and 1,3,5,7,9 would be the
> right cpus to use!

Ok cool thanks Justin...so basically I wanna stagger these out so I 
don't have several processes on the same core ya?

cat /proc/cpuinfo  | egrep "processor|core id"
processor       : 0
core id         : 0
processor       : 1
core id         : 0
processor       : 2
core id         : 1
processor       : 3
core id         : 1
processor       : 4
core id         : 2
processor       : 5
core id         : 2
processor       : 6
core id         : 3
processor       : 7
core id         : 3
processor       : 8
core id         : 4
processor       : 9
core id         : 4
processor       : 10
core id         : 5
processor       : 11
core id         : 5
processor       : 12
core id         : 0
processor       : 13
core id         : 0
processor       : 14
core id         : 1
processor       : 15
core id         : 1
processor       : 16
core id         : 2
processor       : 17
core id         : 2
processor       : 18
core id         : 3
processor       : 19
core id         : 3
processor       : 20
core id         : 4
processor       : 21
core id         : 4
processor       : 22
core id         : 5
processor       : 23
core id         : 5

1,3,5,7,9,11 seem to be the best ones here.  Thanks...that's super 
helpful!

James