Which exception can be generated when subtracting ESP or RSP register? (stack growing)

Question

I'm trying to understand how exactly memory pages for stack is allocated/assigned.

I wrote the following proof-of-concept C-code which obviously causes segmentation fault (on x86_64 Linux):

#include <string.h>

int main()
{
    char a;

    memset( (&a - 4444444), 0, 3333333 );

    return 0;
}

The following fragment of assembly code (AT&T syntax) is generated by gcc from above C-program:

subq    $16, %rsp
leaq    -1(%rbp), %rax
subq    $4444444, %rax
movl    $3333333, %edx
movl    $0, %esi
movq    %rax, %rdi
call    memset

If I add subq $5555555, %rsp manually before calling memset:

subq    $16, %rsp
leaq    -1(%rbp), %rax
subq    $4444444, %rax
movl    $3333333, %edx
movl    $0, %esi
movq    %rax, %rdi
subq    $5555555, %rsp /* added manually */
call    memset

Then segmentation fault disappears because virtual memory pages for stack was assigned after subtracting rsp register caused some hardware exception and assigned exception handler was called (of course, in kernel space).

I know that calling memset here will cause "minor page fault" exceptions. But it's a different story (i.e. allocating physical memory pages).

My question is: Which exception was generated when subq $5555555, %rsp is invoked? I suggest it would be "stack fault" exception but I did not find exact proof for it.

x86 doesn't even have these registers... perhaps you used the wrong architecture tag? — Ben Voigt
@Ben Voigt I have Intel® Core™ i3-2328M CPU @ 2.20GHz × 4, so it's x86 registers. — uintptr_t
RSP is not an x86 register... it's available on your CPU only in long mode, which is not x86. — Ben Voigt

uintptr_t uintptr_t · Accepted Answer · 2015-01-11T13:44:16

I figured it out. First of all, subtracting rsp register does nothing. Second, when we try to write to non-mapped stack area "minor page fault" exception handler is invoked in kernel space. Then this page fault handler checks whether it was legal write or non-legal. I think page fault handler compares with current stack pointer of the thread (in our case it's saved value rsp register). If address where process try to write is upper than current stack pointer then page fault handler expand process's virtual address space and maps this virtual page to physical page, otherwise handler sends SIGSEGV to the process.

I examined the following fragment by using GDB and /proc/[pid]/maps:

subq    $1500016, %rsp
movq    %fs:40, %rax
movq    %rax, -8(%rbp)
xorl    %eax, %eax
movb    $44, -1500016(%rbp)
movb    $55, -1100016(%rbp)
movb    $66, -600016(%rbp)

When subq $1500016, %rsp is invoked stack address range isn't changed. But when first write happens by movb $44, -1500016(%rbp), stack address range is expanded as I explained above.

Which exception can be generated when subtracting ESP or RSP register? (stack growing)

2 Answers