Skip to content

hugepages.md

Latest commit

 

History

History
98 lines (78 loc) · 5.04 KB

hugepages.md

File metadata and controls

98 lines (78 loc) · 5.04 KB
Raw

Backing Guest Memory by Huge Pages

Firecracker supports three modes for the huge_pages field of PUT or PATCH requests to the /machine-config endpoint:

  • None (default): Uses the host's default page size (typically 4K) with no huge page behavior.
  • Transparent: Uses madvise(MADV_HUGEPAGE) to request transparent huge pages for guest memory. Guest memory size must be a multiple of 2MB.
  • 2M: Backs guest memory by 2MB hugetlbfs pages.

Transparent Huge Pages (THP)

Setting huge_pages to Transparent enables transparent huge pages for guest memory via madvise(MADV_HUGEPAGE). This allows the kernel to opportunistically back guest memory with huge pages without requiring a pre-allocated hugetlbfs pool.

Note that while traditional THP uses PMD-sized pages (2MB on x86_64), the actual THP size depends on the CPU architecture. Modern kernels also support "multi-size THP" (mTHP), which can allocate pages in various power-of-2 sizes between the base page size and PMD size (e.g. 16K, 32K, 64K). Firecracker requires guest memory size to be a multiple of 2MB regardless of the THP size used by the host kernel.

Limitations:

  • When vhost-user-blk devices are in use, guest memory is memfd-backed (shared memory). THP for shared/shmem memory is controlled separately from anonymous memory via /sys/kernel/mm/transparent_hugepage/shmem_enabled and may not be enabled by default. Refer to the Linux Documentation on shmem THP for details on how to configure it.
  • THP does not integrate with UFFD; no transparent huge pages will be allocated during userfault-handling while resuming from a snapshot.

Please refer to the Linux Documentation for more information.

Hugetlbfs (2M)

Setting huge_pages to 2M backs guest memory by 2MB hugetlbfs pages. This can bring performance improvements for specific workloads, due to less TLB contention and less overhead during virtual->physical address resolution. It can also help reduce the number of KVM_EXITS required to rebuild extended page tables post snapshot restore, as well as improve boot times (by up to 50% as measured by Firecracker's boot time performance tests)

Using hugetlbfs requires the host running Firecracker to have a pre-allocated pool of 2M pages. Should this pool be too small, Firecracker may behave erratically or receive the SIGBUS signal. This is because Firecracker uses the MAP_NORESERVE flag when mapping guest memory. This flag means the kernel will not try to reserve sufficient hugetlbfs pages at the time of the mmap call, trying to claim them from the pool on-demand. For details on how to manage this pool, please refer to the Linux Documentation.

Huge Pages and Snapshotting

The huge_pages field on PUT /snapshot/load selects the page configuration for the restored microVM. Snapshot reuses the configuration serialized in the snapshot and is the default when the field is omitted. None uses the host's default memory-mapping behavior, while Transparent and 2M select their corresponding huge page configurations.

Explicit 2M hugetlbfs pages require UFFD, so combining 2M with file-backed restore returns an error. Transparent is accepted with UFFD, although UFFD may limit the effectiveness of transparent huge pages.

When restoring snapshots via UFFD, Firecracker will send the configured page size (in KiB) for each memory region as part of the initial handshake, as described in our documentation on UFFD-assisted snapshot-restore.

Known Limitations

Enabling dirty page tracking for hugepage memory negates the performance benefits of using huge pages. This is because KVM will unconditionally establish guest page tables at 4K granularity if dirty page tracking is enabled, even if the host uses huge mappings.

The traditional balloon cannot free hugetlbfs memory. A huge page can only be freed whole, but Linux guests inflate the traditional balloon at most 1 MiB at a time. Firecracker zeroes the inflated pages rather than freeing them. If the huge page holding an inflated page was not backed yet, zeroing takes it from the pool, so inflating can increase host memory use. Use free page hinting or free page reporting instead. Free page reporting needs the guest to report ranges of at least 2 MiB, which Linux guests do by default. See the balloon documentation for details.

Freeing anonymous hugetlbfs memory needs a host kernel of 5.18 or later. Older kernels, such as 5.10, reject MADV_DONTNEED on hugetlbfs mappings, so Firecracker cannot free this memory on those hosts. Hugetlbfs memory shared with vhost-user devices is not affected, because Firecracker frees it with MADV_REMOVE.