mmap
(when used with MAP_ANONYMOUS
) allocates a chunk of RAM that can be placed anywhere within the process's virtual address space, and that can be deallocated later (with munmap
) independently of all other allocations.
brk
changes the ending address of a single, contiguous "arena" of virtual address space: if this address is increased it allocates more memory to the arena, and if it is decreased, it deallocates the memory at the end of the arena. Therefore, memory allocated with brk
can only be released back to the operating system when a continuous range of addresses at the end of the arena is no longer needed by the process.
Using brk
for small allocations, and mmap
for big allocations, is a heuristic based on the assumption that small allocations are more likely to all have the same lifespan, whereas big allocations are more likely to have a lifespan that isn't correlated with any other allocations' lifespan. So, big allocations use the system primitive that lets them be deallocated independently from anything else, and small allocations use the primitive that doesn't.
This heuristic is not very reliable. The current generation of malloc
implementations, if I remember correctly, has given up altogether on brk
and uses mmap
for everything. The malloc
implementation I suspect you are looking at (the one in the GNU C Library, based on your tags) is very old and mainly continues to be used because nobody is brave enough to take the risk of swapping it out for something newer that will probably but not certainly be better.