What prevents an user application from being able to "hijack" into kernel mode?

0
votes

From my understanding. kernel mode is a hardware feature. Ex. it can be set via a register (value1 -> kernel mode, value2 -> user mode).

When the kernel loads and runs an user application, the user application should communicate to the kernel via system call to perform privileged action, during which an interrupt will happen, the execution will switch to kernel mode and the privileged action performed.

My question is:

What is the mechanism that prevents a malicious user application from setting that "mode" register and enter the kernel mode (ex. for x86)? It make sense that only the kernel can set this register, I would like to know more details about how this is enforced.

1
operating-systemkernel
"It make sense that only the kernel can set this register" - Exactly that. Any architecture (which has user/kernel separation) prevent unprivileged code from modifying "mode" register. "I would like to know more details about how this is enforced." - Just take a specific architecture and read documentation for its registers model and instructions set. – Tsyvarev
"Just take a specific architecture and read documentation for its registers model and instructions set". Thanks, that's what I am looking for. I read some textbooks but none contains such info – user2628641
User manual and assembler manual for your processor/MMU. – Martin James

1 Answers

1
votes

I don't know about how this is enforced in hardware itself. It also depends on the architecture. In software for x86, it depends because there are several entry points. When the CPU boots, it is in kernel mode. It can execute every instruction and do whatever it pleases with main memory.

The kernel will thus take advantage of this to set up the page tables and the interrupt handlers during boot before starting any user mode processes.

On x86, kernel mode vs user mode is enforced by the page tables. If a user mode process attempts to access a page which is set as kernel mode it will trigger a fault and call an interrupt handler in kernel mode. The kernel will thus kill the process.

Interrupts are not meant to be an entry point to the kernel. They can still be if a fault happens but then the user mode process won't know and it will sometimes kill the process (if the kernel decides it should).

On x86, the real entry point to the kernel is the LSTAR MSR register. This register can be set from kernel mode only. It can be used alongside the syscall instruction in assembly to jump to the address specified in the register. User mode processes cannot jump in the kernel unless they use the syscall instruction. It thus allows the kernel to set up some services for user mode that are called system calls.