Linux ioctl return value interpreted by who?

Question

I'm working with a custom kernel char device which sometimes returns large negative values (around the thousands, say -2000) for its ioctl().

In userspace, I don't get these values returned from the ioctl call. Instead I get a return value of -1 back with errno set to the negated value from the kernel module (+2000).

As far as I can read and google, __syscall_return() is the macro which is supposed to interpret negative return values as errors. But, it only seems to look for values between -1 and -125. So I didn't expect these large negative values to be translated.

Where are these return values translated? Is it expected behaviour?

I am on Linux 2.6.35.10 with EGLIBC 2.11.3-4+deb6u6.

Answer 1

The translation and move to errno occur on the libc level. Both Gnu libc and μClibc treat negative numbers down to at least -4095 as error conditions, per http://www.makelinux.net/ldd3/chp-6-sect-1

See https://github.molgen.mpg.de/git-mirror/glibc/blob/85b290451e4d3ab460a57f1c5966c5827ca807ca/sysdeps/unix/sysv/linux/aarch64/ioctl.S for the Gnu libc implementation of ioctl.

Answer 2

So, with the help of BRPocock I will report my findings here.

The linux kernel will do a error check for all syscalls along the lines of (from unistd.h):

#define __syscall_return(type, res) \
do { \
        if ((unsigned long)(res) >= (unsigned long)(-125)) { \
                errno = -(res); \
                res = -1; \
        } \
        return (type) (res); \
} while (0)

Libc will also do an error check for all syscalls along the lines of (from syscall.S):

    .text
ENTRY (syscall)

    PUSHARGS_6      /* Save register contents.  */
    _DOARGS_6(44)       /* Load arguments.  */
    movl 20(%esp), %eax /* Load syscall number into %eax.  */
    ENTER_KERNEL        /* Do the system call.  */
    POPARGS_6       /* Restore register contents.  */
    cmpl $-4095, %eax   /* Check %eax for error.  */
    jae SYSCALL_ERROR_LABEL /* Jump to error handler if error.  */
    ret         /* Return to caller.  */

PSEUDO_END (syscall)

Glibc gives a reason for the 4096 value (from sysdep.h):

/* Linux uses a negative return value to indicate syscall errors,
unlike most Unices, which use the condition codes' carry flag.
Since version 2.1 the return value of a system call might be
negative even if the call succeeded.  E.g., the `lseek' system call
might return a large offset.  Therefore we must not anymore test
for < 0, but test for a real error by making sure the value in %eax
is a real error number.  Linus said he will make sure the no syscall
returns a value in -1 .. -4095 as a valid result so we can savely
test with -4095.  */

__syscall_return seems to be missing from newer kernels, I haven't researched that yet.