Repository navigation
Conversation
dd92ab9 to
9f140e2
Compare
5223184 to
6bd3352
Compare
Lt-Flash
changed the title
6642c73 to
0c21c59
Compare
|
Hi, This PR is ready for review. It is a single commit on top of current master, and the description has the measurements and the design summary; full documentation and benchmarks are in HG_MALLOC is opt-in ( Thank you for your time! |
9e02cc0 to
8de237b
Compare
|
Two fixes folded into the commit, both in paths the earlier rounds did not exercise:
Checked with the default build and with |
8de237b to
e749a5a
Compare
|
Pushed a small portability fix, squashed into the HG_MALLOC commit. HG_MALLOC is enabled by default in There is no portable way to free shared pages, so off Linux Tested by compiling the HG sources the way a non-Linux build sees them (Linux-only |
|
Updated the description with call-rate data over TLS and WSS (plus UDP) from a second setup. There one unit is a call relayed by a stateful proxy (2 transactions, 13 datagrams), not an INVITE answered by tm as in the existing tables, so its rates do not compare with theirs. 8 workers per transport, Debian 13 / Linux 6.12, no hugetlb pool, both allocators' shared memory on plain 4 KB pages (the host's THP At the same rate of relayed calls, F_MALLOC -> HG_MALLOC:
Over TLS and WSS the saving is in the TCP workers. TCP main, which does the TLS I/O, is within 3% with either allocator and sets the ceiling, so there HG_MALLOC lowers the cost per call rather than raising capacity. Over UDP with the default 512 KB socket buffer the highest sustained rate goes from 10,000 to 20,000 relayed calls/s (22,000 with Edited 2026-10-19: on that host HG_MALLOC's shared arena ran on 4 KB pages, not on THP as first stated; added the units and the 8 MB socket-buffer result. |
e749a5a to
31d1c97
Compare
|
While going through the allocator again for this round, I found a number of issues and fixed them. I have kept the fixes as a separate commit on top of the original one, so what changed is easy to review on its own; the tests and the reviewer section of the README follow as two more commits. Fixes (
Tests ( For reviewers: a guide to the code ( What changes for users who do not select HG_MALLOC:
Questions:
|
|
For reviewers: a guide to the code ( |
|
I re-ran the benchmarks from the description against the current head ( Setup A, INVITEs answered by tm, mean of two runs:
Registrar, 1.2 M REGISTERs at 60k/s: HG_MALLOC again registers all of them with a p99 of 4 ms on 1.9 cores, against 15-24 s on 4 cores for F_MALLOC and F_PARALLEL_MALLOC and 0.53 M registrations for HP_MALLOC. The binary behind the description's tables, run alongside, is within 3.2% of the current head in CPU, in both directions. Setup B, calls relayed by a stateful proxy, F_MALLOC vs HG_MALLOC in the three receive modes of #4365:
All tables, the method and what differed between the runs: https://github.com/Lt-Flash/opensips/blob/hg_malloc/docs/benchmarks-after-review.md |
|
I pushed six more commits. They come from an internal review of the allocator that I did just before opening this PR; they were still being tested then, so I left them out. They have been running in production since 17 October, on SBCs, registrars and load balancers, and I'm adding them so that this PR is the code that runs there:
Each commit message has the details and the measurements. New head: |
|
I trimmed the comments, fixed one bug and split the large files into smaller ones to make the code easier to read and review. The moved code is unchanged, function by function; six static helpers that other files now call gained an
The full map is in |
|
In 52ac3f3 I added an optional integrity level,
A guarded cell is marked by its own state value, which the header checksum covers, so the level can change while cells of both kinds are live: the shared arena starts guarding as soon as the config is read. The header layout is unchanged on every ABI; a 32-bit build with only one of DBG_MALLOC and SHM_EXTRA_STATS has no spare word and keeps the size in the cell's last 4 bytes instead. Measured on one 16-vCPU host (sipp on the same host, opensips and sipp pinned to separate CPUs, default
With a148e3e fixes the shrink decision. It asked for a whole grow step of free space plus four reserve floors (two with a profile), the floor taken at the current size, while the drain itself releases page by page and stops before any page that would leave less free space than twice the floor of what remains. So an arena under about a step plus four floors never shrank: a pkg arena that grew once from a 2 MB INIT, to 18 MB with the default 16 MB step, stayed at 18 MB. The decision now asks for what the drain tick asks for, one page plus the same multiple of the floor. On the same host, two SIP workers ( |
|
Three more commits:
The clang-arm64-qemu-cross failure in the previous run was in installing the build dependencies (ldconfig crashed under qemu), before anything was built. |
|
166d998 fixes a mistake in 9412ca7: internal_fork() returns the child's slot in the process table, not its pid, so the wait passed the slot to waitid(), found no such child and reported every child as having exited without a verdict - the red run on 9412ca7. The pid is now read from pt[], where the child stores it before its start-up completes. On 166d998 every unit-test job passes, the four qemu cross-builds included, with the fuzz seeds taking 8-19 s there. The one remaining failure, gcc-arm64-qemu-cross in the multi-architecture build, is again ldconfig crashing under qemu while the dependencies install, before anything is built. |
|
Hi @bogdan-iancu, I saw you tagged this as investigating. Before I pile more onto it, what do you think: is a new allocator something you'd take into core? I know it's a lot more complex than the existing allocators, with a lot more functionality. I don't think it's over-engineered, though: sometimes getting the performance out of something takes a more involved design. Earlier versions have been running in production on my servers since early August, and the current one on more than twenty of them for the last few weeks, so I'm fairly confident in it. It's also big (about 6.8k lines of code), so I'm happy to break it into separate PRs: a fixed-size arena first (about 4k lines), then one each for growth, shrink, debugging and stats. Would that work? I'll keep this as a draft until then. |
743d463 to
78f4726
Compare
|
Rebased onto current master (7a4e46b). There were no conflicts, and the rebase changed nothing in the diff. One fix has gone in since the description was last updated: in an LXC container (Proxmox, LXD) |
78f4726 to
6293c4a
Compare
|
This is now ready for review. I'd appreciate any feedback when you have time, thank you! |
6293c4a to
5a272c9
Compare
init_unit_tests() runs at the start of init_modules() and allocates the global context there, before any mod_init() registers its context slots. A test that then uses a module's slot writes past the end of that buffer: the dialog test does, with ctx_dialog_set(), and Q_MALLOC_DBG aborts with "beginning overwritten" on arm32.
A new shared- and private-memory allocator, selected at runtime with -a HG_MALLOC (or -s/-k for one memory type); no behaviour change for existing configurations. - 21 size classes (64 B - 64 KB) served from per-process free lists, so the common allocation and free take no lock; cells move between processes through a shared per-class pool, in batches. Larger requests go to a coalescing large tier. - A transfer cache in each shared arena parks a donated private chain whole and hands it whole to a refill, a pointer swap under the arena lock instead of a per-cell walk; up to 64 chains per class, 1/128 of the arena and at most 32 MB. An idle class gives one chain back to its blocks per GC cycle; a carve at exhaustion drains them all. - Blocks are carved from a buddy allocator over a grid of pages the size of the host's huge page (2 MB on x86-64; 256 KB for small arenas on 4 KB memory), whichever tier backs them; a block whose cells are all free returns to the buddy and can be re-cut for any class. - The arena is acquired through a four-tier ladder - MAP_HUGETLB, THP via MADV_HUGEPAGE, MADV_COLLAPSE, plain 4 KB pages - each tier verified from the kernel's counters. Huge pages are an optimisation, not a requirement. MAP_HUGETLB is for shared arenas only; a private arena starts on THP or 4 KB pages. A refused MAP_HUGETLB is logged with the pool's free pages per NUMA node. On tier 3 a grow done by a worker that ran out of memory is published on 4 KB pages at once and the maintenance process collapses it afterwards, so no worker waits for MADV_COLLAPSE. - Elastic arenas: -m INIT:CAP / -M INIT:CAP (k/m/g suffixes, bare numbers in MB) reserve the cap and commit pages on demand, in a configurable granule (shm_grow_granule / pkg_grow_granule), growing on exhaustion or ahead of it through an auto_scaling_profile (shm_/pkg_auto_scaling_profile), and releasing quiet pages. A maintenance process grows ahead of demand and runs the release with the arena lock dropped. Growth stops at the cap, the backing tier and a host free-memory floor (hg_ram_floor_mb); a sustained refusal by the backing or the floor latches GROW-BLOCKED, raised as E_CORE_SHM_GROW_BLOCKED (reaching the cap is only a notice). - Host memory is checked before anything is committed, on every tier (MemAvailable above hg_ram_floor_mb, the memory cgroup's limit, plus the hugetlb pool for arenas on hugetlb pages): at startup for every process's -M INIT (refuse when the startup processes do not fit, warn for the auto-scaling maximum and for caps beyond MemTotal), before each fork (an auto-scaler fork that does not fit is skipped), and on each arena creation and grow. - On hugetlb pages every grow is an mlock() of the new range, and mlock() checks RLIMIT_MEMLOCK there as well: the arena warns at startup (and under -C) when the limit cannot cover its growth, and a refused grow names the resource that ran out - the limit or the hugetlb pool. On 4 KB and THP pages a limit too low to pin the arena leaves it running and growing unpinned (swappable) instead. - Every cell header carries the cell's state and a check over its address, class and state; a free moves it from live to free in one compare-and-swap, so a double free is refused, also from two processes at once, and a damaged header is refused, not followed. A per-block free map at the pool boundary stops a free-list walk at a scribbled link. Each is counted and logged and the process keeps running. hg_mem_check = 1 adds a tail guard checked at free and realloc, 2 also aborts on the first corruption. HG_MALLOC_DBG adds red zones and a quarantine of freed cells. ASan or Valgrind (-DHG_VALGRIND) is told which bytes of a private arena a caller may touch, ThreadSanitizer about the arena lock. - core:hg_stats (MI) per arena: tier, committed/reserved, per-class carve and reuse, grows/shrinks, the last grow refusal and its cause, whether the arena runs unpinned, transfer-cache puts/gets and parked chains, failed allocations (alloc_failed, out_of_memory), lock hold/wait histograms per reason and the corruption counters; core:hg_advise (MI) recommends -m/-M from the peaks reached; an hgmem: statistics group, including hg_pkg_grow_refused and hg_pkg_grow_blocked summed over every process's private arena; E_CORE_HG_LOCK_STALL for a hold above hg_lock_stall_us. - mem/mem_arena.h: a module may ask for its own HG-managed arena, independent of the core allocator choice (stubs returning NULL when HG is not built). - Unit tests (mem/test) for the allocator and its pkg/shm integration, fork, double free, corruption and growth cases; CI runs the unit tests with HG_MALLOC and HG_MALLOC_DBG as well. - modules/hgstress: a soak and fault-injection test module, excluded from the default build. Every worker churns stamped shm cells across all classes and the large tier and verifies the stamps while the others churn; MI commands park and release memory to drive grow and shrink, and inject a double free, an overrun or a scribbled free-list link. - PREFERRED_MALLOC=HG_MALLOC makes HG_MALLOC the build's default. - A fix to existing code: add_stats_group() sized a group's array as sizeof(stat_var) * no + 1, giving the first stat a 1-byte array. Design, configuration, deployment and test procedures: mem/README.hg_malloc.md.
The set was (address >> 3) % timer_sets. HG_MALLOC hands out cells of one size class at a stride that is a multiple of 64 bytes from 128 bytes up, so every transaction got the same set and timer_partitions had no effect. A Fibonacci multiply spreads the address bits over all sets.
The bucket was address & (size - 1). Every allocator aligns a dialog to at least 8 bytes, so with the default 16 buckets only 2 were used, and under HG_MALLOC, whose same-size cells sit 64 bytes apart or more, only one: all dialogs of such a profile shared one lock. A Fibonacci multiply spreads them.
5a272c9 to
e6af534
Compare
Summary
HG_MALLOC is a new allocator for OpenSIPS shared and private memory. With today's allocators every
shm_malloc()andshm_free()takes a lock - the one global shared-memory lock with F_MALLOC and Q_MALLOC, one of several with F_PARALLEL_MALLOC and HP_MALLOC - so at high call rates the workers queue behind each other. HG_MALLOC's common allocation and free take no lock at all. It is opt-in: nothing changes unless-a HG_MALLOCis given or the build setsPREFERRED_MALLOC=HG_MALLOC.-m 128:1024 -M 16:64- the shared arena and every process's private arena start small, grow with the traffic and give memory back when it goes quiet; growth stops at a host free-memory floor and the memory cgroup's limit, not at the OOM killer.shm_auto_scaling_profileandpkg_auto_scaling_profile, in the grammar of the worker autoscaler'sauto_scaling_profile, grow an arena before an allocation fails and shrink it after a quiet spell.-m/-Mthen only set the starting size: the profile'sscale down tomay take the arena below it and itsscale up toabove it, as far as the cap --m 128:1024withscale up to 1024andscale down to 64starts at 128 MB, grows to 1 GB under load and drops to 64 MB when quiet.-a HG_MALLOC_DBGadds red zones and a quarantine for freed memory.core:hg_stats,core:hg_advise(what-m/-Mshould be),hgmem:statistics and two events.Details
Both tests ran on one bare-metal server (2 x Xeon E5-2690 v2) with 8 UDP workers, the default socket buffer and a fixed 64 GB shared arena on 4 KB pages for every allocator - only the allocator changes - and sipp on four VMs. Two 30 s runs per rate, the allocators taking turns; no run logged an error in OpenSIPS.
INVITEs answered by tm - sipp sends an INVITE, tm answers 486 Busy Here, sipp ACKs - at 60,000 calls/s; in brackets the difference from F_MALLOC:
HG_MALLOC on huge pages used a fixed 12 GB arena on the host's hugetlb pool. At 120,000 calls/s, the highest rate tried, HG_MALLOC still held on both page sizes (2 failed calls in 14.4 million); HP_MALLOC fails from 75,000, F_MALLOC from 90,000 and F_PARALLEL_MALLOC in one of two runs at 120,000.
Calls relayed by OpenSIPS as a record-routing proxy (sipp -> OpenSIPS -> sipp: INVITE, 180, 200, ACK, BYE, 200) - the highest rate at which at most 0.1% of the calls failed:
The same relayed calls over TLS and over WSS, carried to the server by an OpenSIPS gateway on each sipp VM - the highest rate held in both runs, with no error in the OpenSIPS or gateway logs:
Where both held, HG_MALLOC used 14-23% fewer cores than the other allocator, 30-38% less in its TCP workers. At 16,000 calls/s, held by neither, it failed 0.07-0.19% of the calls over TLS and 6.6-9.2% over WSS, F_PARALLEL_MALLOC 8.8-9.6% and 42.9-47.8%.
How each number was measured, and every run:
Solution
timer_partitionsand those profiles spread their load under HG_MALLOC.Design, configuration and a guide for reviewers:
mem/README.hg_malloc.md; docs and all measurements: https://github.com/Lt-Flash/opensips/tree/hg_mallocCompatibility
-a HG_MALLOC, or as the default of a build withPREFERRED_MALLOC=HG_MALLOC.-m/-Malso accept k/m/g suffixes andINIT:CAP; only HG_MALLOC uses the cap.LimitMEMLOCK=infinityis recommended, so the arena stays pinned.add_stats_group()gave a group's first statistic a 1-byte array, and the unit-test runner allocated the global context beforemod_init(), so a module test that sets a module's context slot wrote past it (the first commit; the dialog test aborts on arm32). The other two fixes this PR carried, the default allocator of a build withoutDBG_MALLOCand theF_PARALEL_MALLOCtypo inshm_mem.c, are now in master on their own.Closing issues
None.