As far as I know, trap is something that happens when special events occur. In case of a system call, program executes trap instruction and jumps to kernel mode. Then, trap handler jumps to desired handler (e.g. fork, exec, open).
When the execution of fork, exec, open, etc. is finished, the OS calls return-from-trap instruction and makes the program go back to user mode.
But what exactly is a trap handler? (Also, if you may, what is a trap table?)
The trap handler is the code that will run when the trap is triggered. In your example, the OS will have installed a handler (i.e. told the CPU a memory address of code to run when the trap happens), and the handler will execute the system call. It is NOT the program that jumps to kernel mode. The program is interrupted immediately after it triggers the trap. Execution resumes with the trap handler.
This way, the three layers (program that runs in protected mode, operating system that runs in privileged mode and CPU/hardware that enforces that currently executing code cannot break out of protected mode) can hand off control between each-other.
Also note that a) modern CPU have dedicated instructions for system calls -- a mechanism that is more efficient than trapping but conceptually works the same and that b) there are other traps/interrupts used for different purposes as well -- they provide the basic mechanism for stopping sequential program execution and do "something else" in reaction to some kind of event.
Read this, http://www.cse.iitd.ernet.in/~sbansal/os/lec/l8.html
The x86 processor uses a table known as the interrupt descriptor table (IDT) to determine how to transfer control when a trap occurs. The x86 allows up to 256 different interrupt or exception entry points into the kernel, each with a different interrupt vector. A vector is a number between 0 and 256. An interrupt's vector is determined by the source of the interrupt: different devices, error conditions, and application requests to the kernel generate interrupts with different vectors. The CPU uses the vector as an index into the processor's IDT, which the kernel sets up in kernel-private memory of the kernel's choosing, much like the GDT. From the appropriate entry in this table the processor loads:
