Reservation of hugepages on a Linux system

Question

My VMWare guest system details:

Linux 2.6.32-358.el6.x86_64 (RH 6.4 - Santiago)

# cat /proc/meminfo
MemTotal:        8058796 kB
MemFree:         5145692 kB
Buffers:           32320 kB
Cached:           291312 kB
SwapCached:            0 kB
Active:          1524652 kB
Inactive:         192444 kB
Active(anon):    1393628 kB
Inactive(anon):     1196 kB
Active(file):     131024 kB
Inactive(file):   191248 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       4063224 kB
SwapFree:        4063224 kB
Dirty:               144 kB
Writeback:             0 kB
AnonPages:       1393488 kB
Mapped:            47288 kB
Shmem:              1364 kB
Slab:              52080 kB
SReclaimable:      18572 kB
SUnreclaim:        33508 kB
KernelStack:        3776 kB
PageTables:        15864 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     4063224 kB
Committed_AS:    3101408 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      160248 kB
VmallocChunk:   34359572656 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
HugePages_Total:     512
HugePages_Free:      240
HugePages_Rsvd:      240
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       10240 kB
DirectMap2M:     8378368 kB

grub contents:
transparent_hugepage=never default_hugepagesz=2M hugepagesz=2M hugepages=512

# sysctl -a | grep vm
vm.overcommit_memory = 0
vm.panic_on_oom = 0
vm.oom_kill_allocating_task = 0
vm.extfrag_threshold = 500
vm.oom_dump_tasks = 1
vm.would_have_oomkilled = 0
vm.overcommit_ratio = 0
vm.page-cluster = 3
vm.dirty_background_ratio = 10
vm.dirty_background_bytes = 0
vm.dirty_ratio = 20
vm.dirty_bytes = 0
vm.dirty_writeback_centisecs = 500
vm.dirty_expire_centisecs = 3000
vm.nr_pdflush_threads = 0
vm.swappiness = 60
vm.nr_hugepages = 512
vm.nr_hugepages_mempolicy = 512
vm.hugetlb_shm_group = 0
vm.hugepages_treat_as_movable = 0
vm.nr_overcommit_hugepages = 0
vm.lowmem_reserve_ratio = 256   256     32
vm.drop_caches = 0
vm.min_free_kbytes = 2048
vm.extra_free_kbytes = 0
vm.percpu_pagelist_fraction = 0
vm.max_map_count = 65530
vm.laptop_mode = 0
vm.block_dump = 0
vm.vfs_cache_pressure = 100
vm.legacy_va_layout = 0
vm.zone_reclaim_mode = 0
vm.min_unmapped_ratio = 1
vm.min_slab_ratio = 5
vm.stat_interval = 1
vm.mmap_min_addr = 4096
vm.numa_zonelist_order = default
vm.scan_unevictable_pages = 0
vm.memory_failure_early_kill = 0
vm.memory_failure_recovery = 1

My application needs to grab as many huge pages as it can when it starts, but the fact that HugePages_Free equals HugePages_Rsvd means that it is unable to reserve any.

What is the reason for this, and how do I disable the reservation of hugepages by other applications, if there are any?

Thanks

Andriy Berestovskyy Andriy Berestovskyy · Accepted Answer · 2017-05-26T08:42:49

What is the reason for this, and how do I disable the reservation of hugepages by other applications, if there are any?

The reason for this is that Linux uses Huge Pages for allocations and memory mappings. Please have a look at mmmap(2) (MAP_HUGETLB flag), madvice(2) (MADV_HUGEPAGE flag) and transparent huge pages:

https://www.kernel.org/doc/Documentation/vm/transhuge.txt

So basically the reason for this is that any application running on your system might ask system to use huge pages or even the Linux itself might place your allocations to huge pages transparently.

So to disable all those implicit usages, you might look into /sys/kernel/mm/transparent_hugepage/enabled

On the other hand, you might consider to allocate huge pages right before your application start. Run the system without any huge pages, then configure them with:

echo 512 > /proc/sys/vm/nr_hugepages

And run your application immediately. This will increase the chance the huge pages will be available solely for your application.

Reservation of hugepages on a Linux system

1 Answers