Memory protection in Linux kernel

4
votes

When the shared library is mapped into the memory, Linux kernel will assign virtual memory areas to this memory region and mark their permissions respectively. But we know that there is no executable bit in the page table entry in x86 arch. If there is a call instruction like "call *edx" in the program to invoke function in the shared library, how can the Linux kernel know whether the target address is executable or not? Will it cause general protection fault if the permission is not compatible in the vma list?

2
linuxmemory-managementlinux-kernelx86
a "library" would imply that you're loading executable code, so by definition the memory region the library's loaded into would have to be marked as executable. - Marc B
If PAE is enabled, you can have (no-)execution control. - Alexey Frunze
Before NX support (which works in 32bit mode as well, and Linux uses it), no-exec protection could be done using x86 segment limits. According to that article, Solar Designer's OpenWall distro used it. More importantly, Solar Designer actually invented the technique for non-executable stacks. Apparently it wasn't perfect. I haven't read that exploit to figure out whether that was a fixable problem in implementation, or a fundamental limitation without proper NX - Peter Cordes

2 Answers

3
votes

It can't.

On an architecture (/operating mode) where there is no execute page permission distinct from the read permission, the kernel would not be able to discover a supposedly illegal address execution as a result of a fault detected by the MMU.

0
votes

You're right that in theory the kernel could decide based on the finer-grained permissions on the memory region object, but such a decision making procedure would have to be in the page fault handler, which (I suppose) would make routine memory accesses very expensive.

The kernel instead employs these simplification rules on x86:

  • read access right always implies execute access right
  • write access right always implies read access right

Source: Understanding Linux Kernel, 1st edition, page 205