Why does my program not overflow the stack when I allocate a 11MB char array while the stack upper limit is 10MB?

Question

I have two simple C++ programs and two questions here. I'm working in CentOS 5.2 and my dev environment is as follows:

• g++ (GCC) 4.1.2 20080704 (Red Hat 4.1.2-50)
• "ulimit -s" output: 10240 (kbytes), that is, 10MB

Program #1:

main.cpp:

int main(int argc, char * argv[])
{
    char buf[1024*1024*11] = {0};
    return 0;
}

(Compiled with "g++ -g main.cpp")

The program allocates 1024*1024*11 bytes(that is, 11MB) on the stack but it doesn't crash. After I change the allocation size to 1024*1024*12(that is, 12MB), the program crashes. I think this should be caused by a stack overflow. But Why does the program not crash when the allocation size is 11MB, which is also greater than the 10MB-upper-limit??

Program #2:

main.cpp:

#include <iostream>

int main(int argc, char * argv[])
{
    char buf[1024*1024*11] = {0};

    std::cout << "*** separation ***" << std::endl;

    char buf2[1024*1024] = {0};

    return 0;
}

(Compiled with "g++ -g main.cpp")

This program would result in a program crash because it allocates 12MB bytes on the stack. However, according to the core dump file(see below) the crash occurs on the buf but not buf2. Shouldn't the crash happen to buf2 because we know from program #1 that the allocation of char buf[1024*1024*11] is OK thus after we allocate another 1024*1024 bytes the stack would overflow?

I think there must be some quite fundamental concepts that I didn't build a solid understanding. But what are they??

Appendix: The core-dump info generated by program #2:

Core was generated by `./a.out'.
Program terminated with signal 11, Segmentation fault.
[New process 16433]
#0  0x08048715 in main () at main.cpp:5
5           char buf[1024*1024*11] = {0};

Answer 1

You're wrongly assuming the stack allocations happens where they appear in your code. Anytime you have local variables whose size are known at compile time, space for those will be allocated together when the function is entered. Only dynamic sized local variables are allocated later (VLAs and alloca).

Furthermore the error happens as soon as you write to the memory, not when it's first allocated. Most likely buf is located before buf2 on the stack and the overflow thus happens in buf, not buf2.

Answer 2

To analyze these kind of mysteries, it is always useful to look at the generated code. My guess is that your particular compiler version is doing something different, because mine segfaults with -O0, but not with -O1.

From your program #1, with g++ a.c -g -O0, and then objdump -S a.out

int main(int argc, char * argv[])
{
 8048484:       55                      push   %ebp
 8048485:       89 e5                   mov    %esp,%ebp

This is the standard stack frame. Nothing to see here.

 8048487:       83 e4 f0                and    $0xfffffff0,%esp

Align the stack to multiple of 16, just in case.

 804848a:       81 ec 30 00 b0 00       sub    $0xb00030,%esp

Allocate 0xB00030 bytes of stack space. That is 1024*1024*11 + 48 bytes. No access to the memory yet, so no exception. The extra 48 bytes are of internal use of the compiler.

 8048490:       8b 45 0c                mov    0xc(%ebp),%eax
 8048493:       89 44 24 1c             mov    %eax,0x1c(%esp)   <--- SEGFAULTS

The first time the stack is accessed is beyond the ulimit, so it segfaults.

 8048497:       65 a1 14 00 00 00       mov    %gs:0x14,%eax

Thiis is the stack-protector.

 804849d:       89 84 24 2c 00 b0 00    mov    %eax,0xb0002c(%esp)
 80484a4:       31 c0                   xor    %eax,%eax
    char buf[1024*1024*11] = {0};
 80484a6:       8d 44 24 2c             lea    0x2c(%esp),%eax
 80484aa:       ba 00 00 b0 00          mov    $0xb00000,%edx
 80484af:       89 54 24 08             mov    %edx,0x8(%esp)
 80484b3:       c7 44 24 04 00 00 00    movl   $0x0,0x4(%esp)
 80484ba:       00 
 80484bb:       89 04 24                mov    %eax,(%esp)
 80484be:       e8 d1 fe ff ff          call   8048394 <memset@plt>

Initialize the array, calling memset

    return 0;
 80484c3:       b8 00 00 00 00          mov    $0x0,%eax
}

As you can see, the segfault happens when the internal variables are accessed, because they happen to be below the big array (they have to be, because there is the stack protector, to detect stack smashing).

If you compile with optimizations, the compiler notices that you do nothing useful with the array and optimizes it out. So no sigseg.

Probably your version of GCC is a bit oversmart in non-optimization mode, and removes the array. We can analyze it further if you post the output of objdump -S a.out.

Answer 3

When defining local variables on the stack, there's no real allocation of memory like it is done in the heap. The stack memory allocation consists more simply in changing the address of the stack pointer (that is going to be used by called functions) to reserve the wanted memory.

I suspect that this operation of changing the stack pointer is done only once, at the beginning of the function, to reserve space for all the used local variable (by oposition of changing it once per local variable). This explains why the error on your program #2 occurs on the first allocation.

Answer 4

Both of your programs should ideally give segfault.

Normally whenever a function is entered, all the variables defined in it are allocated memory onto the stack. This said, it however also depends on the optimization level with which the code is compiled.

Optimization level zero, indicated during compilation as -O0 indicates no optimzation at all. It is also the default optimization level with which a code is compiled. The above mentioned programs when compiled with -O0, gives segfault.

However, when you compile the programs using higher optimization levels, the compiler notices that the variable which is defined is not used in the function. It therefore removes the variable definition from the assembly language code. As a result, your programs won't give any segfault.