kernel_optimize_test/mm
Andrea Arcangeli 44518d2b32 mm: tail page refcounting optimization for slab and hugetlbfs
This skips the _mapcount mangling for slab and hugetlbfs pages.

The main trouble in doing this is to guarantee that PageSlab and
PageHeadHuge remains constant for all get_page/put_page run on the tail
of slab or hugetlbfs compound pages.  Otherwise if they're set during
get_page but not set during put_page, the _mapcount of the tail page
would underflow.

PageHeadHuge will remain true until the compound page is released and
enters the buddy allocator so it won't risk to change even if the tail
page is the last reference left on the page.

PG_slab instead is cleared before the slab frees the head page with
put_page, so if the tail pin is released after the slab freed the page,
we would have a problem.  But in the slab case the tail pin cannot be
the last reference left on the page.  This is because the slab code is
free to reuse the compound page after a kfree/kmem_cache_free without
having to check if there's any tail pin left.  In turn all tail pins
must be always released while the head is still pinned by the slab code
and so we know PG_slab will be still set too.

Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>
Reviewed-by: Khalid Aziz <khalid.aziz@oracle.com>
Cc: Pravin Shelar <pshelar@nicira.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Ben Hutchings <bhutchings@solarflare.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Johannes Weiner <jweiner@redhat.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Rik van Riel <riel@redhat.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: Minchan Kim <minchan@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2014-01-21 16:19:43 -08:00
..
backing-dev.c
balloon_compaction.c
bootmem.c
bounce.c
cleancache.c
compaction.c mm/compaction: respect ignore_skip_hint in update_pageblock_skip 2013-12-18 19:04:52 -08:00
debug-pagealloc.c
dmapool.c
fadvise.c
failslab.c
filemap_xip.c
filemap.c mm: drop actor argument of do_generic_file_read() 2013-11-15 09:32:13 +09:00
fremap.c mm: fix use-after-free in sys_remap_file_pages 2014-01-02 14:40:30 -08:00
frontswap.c
highmem.c
huge_memory.c thp: fix copy_page_rep GPF by testing is_huge_zero_pmd once only 2014-01-12 16:47:15 +07:00
hugetlb_cgroup.c
hugetlb.c mm: hugetlb: use get_page_foll() in follow_hugetlb_page() 2014-01-21 16:19:43 -08:00
hwpoison-inject.c
init-mm.c
internal.h mm: tail page refcounting optimization for slab and hugetlbfs 2014-01-21 16:19:43 -08:00
interval_tree.c
Kconfig mm: add missing dependency in Kconfig 2013-12-18 19:04:52 -08:00
Kconfig.debug
kmemcheck.c
kmemleak-test.c
kmemleak.c
ksm.c
list_lru.c
maccess.c
madvise.c
Makefile
memblock.c
memcontrol.c memcg: fix memcg_size() calculation 2014-01-02 14:40:30 -08:00
memory_hotplug.c
memory-failure.c mm/memory-failure.c: transfer page count from head page to tail page after split thp 2014-01-02 14:40:30 -08:00
memory.c dma-debug: introduce debug_dma_assert_idle() 2014-01-21 16:19:41 -08:00
mempolicy.c mm/mempolicy: fix !vma in new_vma_page() 2013-12-18 19:04:52 -08:00
mempool.c
migrate.c aio/migratepages: make aio migrate pages sane 2013-12-21 17:56:08 -05:00
mincore.c
mlock.c mm: munlock: fix deadlock in __munlock_pagevec() 2014-01-02 14:40:30 -08:00
mm_init.c
mmap.c mm: convert mm->nr_ptes to atomic_long_t 2013-11-15 09:32:14 +09:00
mmu_context.c
mmu_notifier.c
mmzone.c
mprotect.c mm: fix TLB flush race between migration, and change_protection_range 2013-12-18 19:04:51 -08:00
mremap.c
msync.c
nobootmem.c
nommu.c Merge branch 'akpm' (patches from Andrew Morton) 2013-11-13 15:45:43 +09:00
oom_kill.c mm: convert mm->nr_ptes to atomic_long_t 2013-11-15 09:32:14 +09:00
page_alloc.c mm: page_alloc: revert NUMA aspect of fair allocation policy 2013-12-20 12:19:18 -08:00
page_cgroup.c
page_io.c
page_isolation.c
page-writeback.c
pagewalk.c
percpu-km.c
percpu-vm.c
percpu.c
pgtable-generic.c mm: fix TLB flush race between migration, and change_protection_range 2013-12-18 19:04:51 -08:00
process_vm_access.c
quicklist.c
readahead.c
rmap.c mm/hugetlb: check for pte NULL pointer in __page_check_address() 2013-12-18 19:04:52 -08:00
shmem.c security: shmem: implement kernel private shmem inodes 2013-12-02 11:24:19 +00:00
slab_common.c
slab.c Merge branch 'slab/next' of git://git.kernel.org/pub/scm/linux/kernel/git/penberg/linux 2013-11-22 08:10:34 -08:00
slab.h
slob.c
slub.c Merge branch 'slab/next' of git://git.kernel.org/pub/scm/linux/kernel/git/penberg/linux 2013-11-22 08:10:34 -08:00
sparse-vmemmap.c
sparse.c
swap_state.c
swap.c mm: tail page refcounting optimization for slab and hugetlbfs 2014-01-21 16:19:43 -08:00
swapfile.c
truncate.c
util.c mm: fix crash when using XFS on loopback 2014-01-15 14:19:42 +07:00
vmalloc.c
vmpressure.c
vmscan.c
vmstat.c
zbud.c
zswap.c