Do any languages / compilers utilize the x86 ENTER instruction with a nonzero nesting level?

Question

Those familiar with x86 assembly programming are very used to the typical function prologue / epilogue:

push ebp ; Save old frame pointer.
mov  ebp, esp ; Point frame pointer to top-of-stack.
sub  esp, [size of local variables]
...
mov  esp, ebp ; Restore frame pointer and remove stack space for locals.
pop  ebp
ret

This same sequence of code can also be implemented with the ENTER and LEAVE instructions:

enter [size of local variables], 0
...
leave
ret

The ENTER instruction's second operand is the nesting level, which allows multiple parent frames to be accessed from the called function.

This is not used in C because there are no nested functions; local variables have only the scope of the function they're declared in. This construct does not exist (although sometimes I wish it did):

void func_a(void)
{
    int a1 = 7;

    void func_b(void)
    {
        printf("a1 = %d\n", a1);  /* a1 inherited from func_a() */
    }

    func_b();
}

Python however does have nested functions which behave this way:

def func_a():
    a1 = 7
    def func_b():
        print 'a1 = %d' % a1      # a1 inherited from func_a()
    func_b()

Of course Python code isn't translated directly to x86 machine code, and thus would be unable (unlikely?) to take advantage of this instruction.

Are there any languages which compile to x86 and provide nested functions? Are there compilers which will emit an ENTER instruction with a nonzero second operand?

Intel invested a nonzero amount of time/money into that nesting level operand, and basically I'm just curious if anyone uses it :-)

References:

+1, the most interesting question of today. For 1), GCC supports nested functions in C using exactly your syntax. But explicitly not in C++. — Iwillnotexist Idonotexist
@IwillnotexistIdonotexist I coincidentally just ran across that same page. Interestingly it compiles on gcc 4.7.2 with the default options. Looking forward to looking at the disassembly. Fun stuff! — Jonathon Reinhart
For what it is worth, I understand from grep-ing gcc-4.8.2/gcc/config/i386/i386.c:10339 that GCC simply never emits ENTER at all nowadays. And the comment at that line is quite clear: /* Note: AT&T enter does NOT have reversed args. Enter is probably slower on all targets. Also sdb doesn't like it. */ — Iwillnotexist Idonotexist
@IwillnotexistIdonotexist FWIW, that was part of the very first version of GCC. git log -p on their cvs->svn->git converted repository shows that it already existed in the initial check-in in 1992. — user743382
And my private svn checkout of LLVM 3.5 has at llvm/lib/Target/X86/X86FrameLowering.cpp:355 a comment for the emitPrologue() method which reads in part ; Spill general-purpose registers [for all callee-saved GPRs] pushq %<reg> [if not needs FP] .cfi_def_cfa_offset (offset from RETADDR) .seh_pushreg %<reg>. There are no mentions of ENTER, only pushes; And the enum constant for x86 ENTER occurs only 3 times in all of LLVM; It doesn't even look as though they have testcases for it. — Iwillnotexist Idonotexist

gsg gsg · Accepted Answer · 2014-10-12T09:27:56

enter is avoided in practice as it performs quite poorly - see the answers at "enter" vs "push ebp; mov ebp, esp; sub esp, imm" and "leave" vs "mov esp, ebp; pop ebp". There are a bunch of x86 instructions that are obsolete but are still supported for backwards compatibility reasons - enter is one of those. (leave is OK though, and compilers are happy to emit it.)

Implementing nested functions in full generality as in Python is actually a considerably more interesting problem than simply selecting a few frame management instructions - search for 'closure conversion' and 'upwards/downwards funarg problem' and you'll find many interesting discussions.

Note that the x86 was originally designed as a Pascal machine, which is why there are instructions to support nested functions (enter, leave), the pascal calling convention in which the callee pops a known number of arguments from the stack (ret K), bounds checking (bound), and so on. Many of these operations are now obsolete.

Do any languages / compilers utilize the x86 ENTER instruction with a nonzero nesting level?

4 Answers