kernel_optimize_test/include/asm-x86_64
akpm@osdl.org 198e2f1811 [PATCH] scheduler cache-hot-autodetect
)

From: Ingo Molnar <mingo@elte.hu>

This is the latest version of the scheduler cache-hot-auto-tune patch.

The first problem was that detection time scaled with O(N^2), which is
unacceptable on larger SMP and NUMA systems. To solve this:

- I've added a 'domain distance' function, which is used to cache
  measurement results. Each distance is only measured once. This means
  that e.g. on NUMA distances of 0, 1 and 2 might be measured, on HT
  distances 0 and 1, and on SMP distance 0 is measured. The code walks
  the domain tree to determine the distance, so it automatically follows
  whatever hierarchy an architecture sets up. This cuts down on the boot
  time significantly and removes the O(N^2) limit. The only assumption
  is that migration costs can be expressed as a function of domain
  distance - this covers the overwhelming majority of existing systems,
  and is a good guess even for more assymetric systems.

  [ People hacking systems that have assymetries that break this
    assumption (e.g. different CPU speeds) should experiment a bit with
    the cpu_distance() function. Adding a ->migration_distance factor to
    the domain structure would be one possible solution - but lets first
    see the problem systems, if they exist at all. Lets not overdesign. ]

Another problem was that only a single cache-size was used for measuring
the cost of migration, and most architectures didnt set that variable
up. Furthermore, a single cache-size does not fit NUMA hierarchies with
L3 caches and does not fit HT setups, where different CPUs will often
have different 'effective cache sizes'. To solve this problem:

- Instead of relying on a single cache-size provided by the platform and
  sticking to it, the code now auto-detects the 'effective migration
  cost' between two measured CPUs, via iterating through a wide range of
  cachesizes. The code searches for the maximum migration cost, which
  occurs when the working set of the test-workload falls just below the
  'effective cache size'. I.e. real-life optimized search is done for
  the maximum migration cost, between two real CPUs.

  This, amongst other things, has the positive effect hat if e.g. two
  CPUs share a L2/L3 cache, a different (and accurate) migration cost
  will be found than between two CPUs on the same system that dont share
  any caches.

(The reliable measurement of migration costs is tricky - see the source
for details.)

Furthermore i've added various boot-time options to override/tune
migration behavior.

Firstly, there's a blanket override for autodetection:

	migration_cost=1000,2000,3000

will override the depth 0/1/2 values with 1msec/2msec/3msec values.

Secondly, there's a global factor that can be used to increase (or
decrease) the autodetected values:

	migration_factor=120

will increase the autodetected values by 20%. This option is useful to
tune things in a workload-dependent way - e.g. if a workload is
cache-insensitive then CPU utilization can be maximized by specifying
migration_factor=0.

I've tested the autodetection code quite extensively on x86, on 3
P3/Xeon/2MB, and the autodetected values look pretty good:

Dual Celeron (128K L2 cache):

 ---------------------
 migration cost matrix (max_cache_size: 131072, cpu: 467 MHz):
 ---------------------
           [00]    [01]
 [00]:     -     1.7(1)
 [01]:   1.7(1)    -
 ---------------------
 cacheflush times [2]: 0.0 (0) 1.7 (1784008)
 ---------------------

Here the slow memory subsystem dominates system performance, and even
though caches are small, the migration cost is 1.7 msecs.

Dual HT P4 (512K L2 cache):

 ---------------------
 migration cost matrix (max_cache_size: 524288, cpu: 2379 MHz):
 ---------------------
           [00]    [01]    [02]    [03]
 [00]:     -     0.4(1)  0.0(0)  0.4(1)
 [01]:   0.4(1)    -     0.4(1)  0.0(0)
 [02]:   0.0(0)  0.4(1)    -     0.4(1)
 [03]:   0.4(1)  0.0(0)  0.4(1)    -
 ---------------------
 cacheflush times [2]: 0.0 (33900) 0.4 (448514)
 ---------------------

Here it can be seen that there is no migration cost between two HT
siblings (CPU#0/2 and CPU#1/3 are separate physical CPUs). A fast memory
system makes inter-physical-CPU migration pretty cheap: 0.4 msecs.

8-way P3/Xeon [2MB L2 cache]:

 ---------------------
 migration cost matrix (max_cache_size: 2097152, cpu: 700 MHz):
 ---------------------
           [00]    [01]    [02]    [03]    [04]    [05]    [06]    [07]
 [00]:     -    19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
 [01]:  19.2(1)    -    19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
 [02]:  19.2(1) 19.2(1)    -    19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
 [03]:  19.2(1) 19.2(1) 19.2(1)    -    19.2(1) 19.2(1) 19.2(1) 19.2(1)
 [04]:  19.2(1) 19.2(1) 19.2(1) 19.2(1)    -    19.2(1) 19.2(1) 19.2(1)
 [05]:  19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)    -    19.2(1) 19.2(1)
 [06]:  19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)    -    19.2(1)
 [07]:  19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)    -
 ---------------------
 cacheflush times [2]: 0.0 (0) 19.2 (19281756)
 ---------------------

This one has huge caches and a relatively slow memory subsystem - so the
migration cost is 19 msecs.

Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Ashok Raj <ashok.raj@intel.com>
Signed-off-by: Ken Chen <kenneth.w.chen@intel.com>
Cc: <wilder@us.ibm.com>
Signed-off-by: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-01-12 09:08:50 -08:00
..
8253pit.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
a.out.h [PATCH] x86_64: TASK_SIZE fixes for compatibility mode processes 2005-06-21 18:46:12 -07:00
acpi.h [ACPI] delete CONFIG_ACPI_PCI 2005-08-25 12:40:44 -04:00
agp.h [PATCH] AGP fix for Xen VMM 2005-06-07 12:35:43 -07:00
apic.h [PATCH] x86_64: Some housekeeping in local APIC code 2006-01-11 19:05:03 -08:00
apicdef.h [PATCH] x86_64: x86_64 write apic id fix 2006-01-11 19:04:57 -08:00
atomic.h [PATCH] x86_64: Make sure BITS_PER_ATOMIC is defined in asm-generic/atomic.h 2006-01-11 19:05:01 -08:00
auxvec.h [PATCH] auxiliary vector cleanups 2005-09-07 16:57:21 -07:00
bitops.h [PATCH] x86_64: fls in asm for x86_64 2006-01-11 19:04:50 -08:00
boot.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
bootsetup.h [PATCH] Increase number of e820 entries hard limit from 32 to 128 2005-05-01 08:58:51 -07:00
bug.h [PATCH] x86-64: reduce x86-64 bug frame by 4 bytes 2005-09-12 10:50:58 -07:00
bugs.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
byteorder.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
cache.h [PATCH] x86_64: Inclusion of ScaleMP vSMP architecture patches - vsmp_align 2006-01-11 19:05:01 -08:00
cacheflush.h [PATCH] x86/x86_64: mark rodata section read-only: x86-64 support 2006-01-06 08:33:36 -08:00
calling.h [PATCH] x86-64: Fix CFI information 2005-09-12 10:50:56 -07:00
checksum.h [NET]: Fix ipl=>ihl typo in ip_fast_csum 2005-08-29 16:02:48 -07:00
compat.h [PATCH] x86_64: Implement is_compat_task the right way 2006-01-11 19:04:53 -08:00
cpu.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
cpufeature.h [PATCH] x86_64: Remove unused AMD K8 C stepping flag 2006-01-11 19:05:02 -08:00
cputime.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
current.h kbuild: alpha,x86_64 use generic asm-offsets.h support 2005-09-09 21:28:48 +02:00
debugreg.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
delay.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
desc.h [PATCH] x86_64: Align and pad x86_64 GDT on page boundary 2006-01-11 19:04:53 -08:00
div64.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
dma-mapping.h [PATCH] x86_64: Use function pointers to call DMA mapping functions 2006-01-11 19:04:55 -08:00
dma.h [PATCH] x86_64: Add 4GB DMA32 zone 2005-11-14 19:55:13 -08:00
dwarf2.h [PATCH] x86_64: Separate CONFIG_UNWIND_INFO from CONFIG_DEBUG_INFO 2006-01-11 19:01:10 -08:00
e820.h [PATCH] kdump: x86_64: add memmmap command line option 2006-01-10 08:01:27 -08:00
elf.h [PATCH] fix remaining missing includes 2005-11-07 07:53:41 -08:00
emergency-restart.h [PATCH] x86_64: Implemenent machine_emergency_restart 2005-07-26 14:35:42 -07:00
errno.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
fcntl.h [PATCH] Clean up struct flock definitions 2005-09-07 16:57:38 -07:00
fixmap.h [PATCH] Replace extern inline with static inline in asm-x86_64/* 2005-09-12 10:50:56 -07:00
floppy.h [PATCH] make some things static 2005-05-05 16:36:47 -07:00
fpu32.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
futex.h [PATCH] FUTEX_WAKE_OP: pthread_cond_signal() speedup 2005-09-07 16:57:17 -07:00
gart-mapping.h [PATCH] x86_64: Use function pointers to call DMA mapping functions 2006-01-11 19:04:55 -08:00
genapic.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
hardirq.h [PATCH] x86-64: Some cleanup and optimization to the processor data area. 2005-09-12 10:49:58 -07:00
hpet.h [PATCH] x86_64: Adjust, correct, and complete the HPET definitions for x86-64. 2005-11-14 19:55:13 -08:00
hw_irq.h [PATCH] x86_64: Remove useless KDB vector 2006-01-11 19:05:00 -08:00
i387.h [PATCH] x86_64: Fix 64bit FXSAVE encoding 2006-01-11 19:04:51 -08:00
ia32_unistd.h [PATCH] Swap Migration V5: sys_migrate_pages interface 2006-01-08 20:12:42 -08:00
ia32.h [PATCH] x86_64: Allow modular build of ia32 aout loader 2005-11-14 19:55:16 -08:00
ide.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
idle.h [PATCH] x86_64: Add idle notifiers 2006-01-11 19:04:55 -08:00
io_apic.h [ACPI] delete CONFIG_ACPI_BOOT 2005-08-24 12:08:54 -04:00
io.h [PATCH] x86_64: Generalize DMI and enable for x86-64 2006-01-11 19:04:51 -08:00
ioctl.h [PATCH] Generic ioctl.h 2006-01-10 08:01:34 -08:00
ioctls.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
ipcbuf.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
ipi.h [PATCH] x86_64: Remove useless KDB vector 2006-01-11 19:05:00 -08:00
irq.h [PATCH] remove unnecessary handle_IRQ_event() prototypes 2005-09-09 13:57:33 -07:00
kdebug.h [PATCH] x86_64: make trap information available to die notification handlers 2006-01-11 19:01:10 -08:00
kexec.h [PATCH] kdump: x86_64 save cpu registers upon crash 2006-01-10 08:01:28 -08:00
kmap_types.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
kprobes.h [PATCH] kprobes: fix build breakage 2006-01-10 08:01:40 -08:00
ldt.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
linkage.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
local.h [PATCH] x86-64: clean up local_add/sub arguments 2005-09-12 10:50:59 -07:00
mach_apic.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
mc146818rtc.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
mce.h [PATCH] x86_64: Support for AMD specific MCE Threshold. 2005-11-14 19:55:13 -08:00
mman.h [PATCH] madvise(MADV_REMOVE): remove pages from tmpfs shm backing store 2006-01-06 08:33:22 -08:00
mmsegment.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
mmu_context.h [PATCH] x86_64: cleanup enter_lazy_tlb() 2006-01-11 19:05:00 -08:00
mmu.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
mmx.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
mmzone.h [PATCH] x86_64: Move NUMA page_to_pfn/pfn_to_page functions out of line 2006-01-11 19:05:01 -08:00
module.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
mpspec.h [PATCH] mpspec: remove unneeded packed attribute 2006-01-06 08:33:39 -08:00
msgbuf.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
msi.h [PATCH] PCI: Change MSI to use physical delivery mode always 2005-11-10 16:09:18 -08:00
msr.h [PATCH] Fix x86_64/msr.h interface to agree with i386/msr.h 2005-11-20 11:52:59 -08:00
mtrr.h [PATCH] Clean up mtrr compat ioctl code 2005-10-30 17:37:13 -08:00
mutex.h Fix mutex_trylock() copy-and-paste bug (x86, x86-64, generic mutex-dec.h) 2006-01-11 15:50:47 -08:00
namei.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
nmi.h [PATCH] x86_64: Collected NMI watchdog fixes. 2005-05-17 07:59:16 -07:00
node.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
numa.h [PATCH] x86_64: Early initialization of cpu_to_node 2006-01-11 19:04:59 -08:00
numnodes.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
page.h [PATCH] x86_64: Move int 3 handler to debug stack and allow to increase it. 2006-01-11 19:01:13 -08:00
param.h [PATCH] Avoid namespace pollution in <asm/param.h> 2006-01-02 08:38:38 -08:00
parport.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
pci-direct.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
pci.h [PATCH] x86_64: Use function pointers to call DMA mapping functions 2006-01-11 19:04:55 -08:00
pda.h [PATCH] x86_64: Allocate PDAs in the local node 2006-01-11 19:04:59 -08:00
percpu.h [PATCH] x86_64: Node local pda take 2 -- cpu_pda preparation 2006-01-11 19:04:59 -08:00
pgalloc.h [PATCH] Replace extern inline with static inline in asm-x86_64/* 2005-09-12 10:50:56 -07:00
pgtable.h [PATCH] x86_64: "extern inline" -> "static inline" in pgtable.h 2006-01-11 19:04:53 -08:00
poll.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
posix_types.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
prctl.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
processor.h [PATCH] x86_64: Inclusion of ScaleMP vSMP architecture patches - vsmp_align 2006-01-11 19:05:01 -08:00
proto.h [PATCH] x86_64: Don't try to synchronize the TSC over CPUs on Intel CPUs at boot. 2006-01-11 19:04:57 -08:00
ptrace.h [PATCH] xen: x86: Rename usermode macro 2005-06-23 09:45:14 -07:00
resource.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
rtc.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
rwlock.h [PATCH] Fix typo in x86_64 __build_write_lock_const assembly 2005-12-24 12:30:22 -08:00
scatterlist.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
seccomp.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
sections.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
segment.h [PATCH] x86_64: Remove unused segments 2006-01-11 19:05:01 -08:00
semaphore.h [PATCH] semaphore: Remove __MUTEX_INITIALIZER() 2005-10-30 17:37:27 -08:00
sembuf.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
serial.h [PATCH] Serial: Split 8250 port table (part 2) 2005-06-29 18:45:19 +01:00
setup.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
shmbuf.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
shmparam.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
sigcontext32.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
sigcontext.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
siginfo.h [PATCH] consolidate SIGEV_PAD_SIZE 2005-05-01 08:59:08 -07:00
signal.h [PATCH] Replace extern inline with static inline in asm-x86_64/* 2005-09-12 10:50:56 -07:00
smp.h [PATCH] x86_64: Fix compile error with !CONFIG_COMPAT 2006-01-11 19:04:57 -08:00
socket.h [NET]: Introduce SO_{SND,RCV}BUFFORCE socket options 2005-08-29 15:31:35 -07:00
sockios.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
sparsemem.h [PATCH] add x86-64 specific support for sparsemem 2005-06-23 09:45:07 -07:00
spinlock_types.h [PATCH] spinlock consolidation 2005-09-10 10:06:21 -07:00
spinlock.h [PATCH] x86_64: Use int operations in spinlocks to support more than 128 CPUs spinning. 2005-11-14 19:55:15 -08:00
stat.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
statfs.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
string.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
suspend.h [PATCH] swsusp: clean assembly parts 2005-06-25 16:24:33 -07:00
swiotlb.h [PATCH] x86_64: Use function pointers to call DMA mapping functions 2006-01-11 19:04:55 -08:00
system.h [PATCH] sched: add cacheflush() asm 2006-01-12 09:08:49 -08:00
termbits.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
termios.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
thread_info.h [PATCH] x86_64: Implement is_compat_task the right way 2006-01-11 19:04:53 -08:00
timex.h [PATCH] x86_64: On Intel CPUs don't do an additional CPU sync before RDTSC 2006-01-11 19:04:58 -08:00
tlb.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
tlbflush.h [PATCH] x86-64: Increase TLB flush array size 2005-09-12 10:49:58 -07:00
topology.h [PATCH] scheduler cache-hot-autodetect 2006-01-12 09:08:50 -08:00
types.h [PATCH] sab: consolidate kmem_bufctl_t 2005-09-05 00:05:48 -07:00
uaccess.h [PATCH] x86_64: Display meaningful part of filename during BUG() 2006-01-11 19:05:03 -08:00
ucontext.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
unaligned.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
unistd.h [PATCH] Swap Migration V5: sys_migrate_pages interface 2006-01-08 20:12:42 -08:00
user32.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
user.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
vga.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
vsyscall32.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
vsyscall.h [PATCH] x86_64: sparse warning cleanups 2006-01-11 19:05:02 -08:00
xor.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00