How to trap system calls in a userspace library in Linux?

0
votes

I have a requirement to write a linux device driver in userspace.

How can I write a library which, when linked to an application, can handle system calls to a particular device.

The application should be able to use open(), read(), write(), ioctl() on a device such as /dev/mydev0, but these calls should terminate in a userspace library instead of a kernel module.

Please advise on if this is possible and how can I achieve this.

1
linuxlinux-kernellinux-device-driver
You can trap the C wrappers of such syscalls using the usual LD_PRELOAD tricks; you cannot trap directly the int 80/sysenter instructions (i.e., if someone calls "manually" the syscall, without going through the libc wrapper) as easily. - Matteo Italia
Hi, Can you please elaborate on how these LD_PRELOAD "tricks" work? That might be something I can use. - Punit Soni
superuser.com/a/370054/25859 - Matteo Italia
Using LD_PRELOAD to trap library calls is exceeding complex, especially if you're trying to capture all IO calls - and that's before you even begin to think about possible multithreaded calling programs. You do not want to go that way unless you have to. Because if a program directly makes system calls, you can't really trap those, as @MatteoItalia has already posted. - Andrew Henle
Following on the LD_PRELOAD tip, I stumbled up-on something similar linux-projects.org/modules/sections/…. A video4linux camera driver framework written in userspace. It also uses LD_PRELOAD to provide some functionality. Unfortunately, the source code for this is not published. I am trying to write something very similar (driver for camera module which provide v4l2 support). - Punit Soni

1 Answers

1
votes

Linux is a monolithic kernel which means in general, what you're asking is not possible; you can't write arbitrary drivers in user-mode.

You could (as your title alludes to), use ptrace(2) to trap on system calls, and basically redirect them to functions in your library. That is not a simple, straightforward solution however.

See also:

FUSE (Filesystem in USErspace) may be what you're looking for. It's a mechanism that allows filesystem drivers specifically to be implemented via a user-space process. This is how sshfs, for example, is implemented.

Resources: