Error while loading a BPF program that copies a buffer to the BPF stack

Question

I am trying to load a BPF program that simply copies the buf parameter of tty_write to the BPF stack. My program is as follows:

#define BUFSIZE 256

SEC("kprobe/tty_write")
int kprobe__tty_write(struct pt_regs *ctx, struct file *file, const char __user *buf, size_t count)
{
  char buffer[BUFSIZE];
  bpf_probe_read(buffer, BUFSIZE, (void *)buf);

  return 0;
}

Note that I am using bpf_helpers.h from tcptracer-bpf to define the SEC macro. In my real program I would actually use buffer for something, but I have not shown that part here. When I try to load the program (from an ELF file using gobpf) I get the following error:

error while loading "kprobe/tty_write" (permission denied):
0: (bf) r1 = r10
1: (07) r1 += -256
2: (b7) r2 = 256
3: (85) call bpf_probe_read#4
R3 !read_ok

Why is this? My program has been adapted from ttysnoop.py so I know it is possible to do what I'm trying to do. The full disassembly of my program is as follows:

Disassembly of section kprobe/tty_write:
kprobe__tty_write:
       0:   bf a1 00 00 00 00 00 00     r1 = r10
       1:   07 01 00 00 00 ff ff ff     r1 += -256
       2:   b7 02 00 00 00 01 00 00     r2 = 256
       3:   85 00 00 00 04 00 00 00     call 4
       4:   b7 00 00 00 00 00 00 00     r0 = 0
       5:   95 00 00 00 00 00 00 00     exit

uname -a: Linux ubuntu1710 4.13.0-32-generic #35-Ubuntu SMP Thu Jan 25 09:13:46 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

EDIT:

As an experiment, I tried loading a program similar to the example program described when bpf_probe_read_str was introduced as a helper function:

#define BUFSIZE 256

SEC("kprobe/sys_open")
void bpf_sys_open(struct pt_regs *ctx)
{
  char buf[BUFSIZE];
  bpf_probe_read(buf, sizeof(buf), (void *)ctx->di);
}

Which loads with no issues and gives the following assembly:

Disassembly of section kprobe/sys_open:
bpf_sys_open:
       0:   79 13 70 00 00 00 00 00     r3 = *(u64 *)(r1 + 112)
       1:   bf a1 00 00 00 00 00 00     r1 = r10
       2:   07 01 00 00 00 ff ff ff     r1 += -256
       3:   b7 02 00 00 00 01 00 00     r2 = 256
       4:   85 00 00 00 04 00 00 00     call 4
       5:   95 00 00 00 00 00 00 00     exit

So it seems that my tty_write program is passing the third register straight to the call to bpf_probe_read from when it was set after the kprobe was triggered; this may be the cause of the error I'm seeing, but I am not sure.

pchaigno pchaigno · Accepted Answer · 2018-02-10T18:50:11

As you've discovered yourself, the issue comes from the use of additional parameters to kprobe__tty_write. This works in ttysnoop because it uses bcc to compile and load BPF programs. bcc actually rewrites the additional parameters to the ctx->xx dereferences. You can see this with the following snippet:

from bcc import BPF
BPF(text="""
#include <linux/ptrace.h>
int kprobe__tty_write(struct pt_regs *ctx, struct file *file, const char __user *buf, size_t count) {
return 0;
}
""", debug=4)

Thanks to debug=4, we can see the rewrites when executing:

$ sudo python tmp.py
clang -cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -emit-llvm-uselists -disable-free -disable-llvm-verifier -discard-value-names -main-file-name main.c -mrelocation-model static -mthread-model posix -fmath-errno -masm-verbose -mconstructor-aliases -fuse-init-array -target-cpu x86-64 -momit-leaf-frame-pointer -dwarf-column-info -debugger-tuning=gdb -coverage-notes-file /usr/src/linux-headers-4.4.0-112-generic/main.gcno -nostdsysteminc -nobuiltininc -resource-dir lib/clang/5.0.1 -isystem /virtual/lib/clang/include -include ./include/linux/kconfig.h -include /virtual/include/bcc/bpf.h -include /virtual/include/bcc/helpers.h -isystem /virtual/include -I /home/paul/bcc2 -I ./arch/x86/include -I arch/x86/include/generated/uapi -I arch/x86/include/generated -I include -I ./arch/x86/include/uapi -I arch/x86/include/generated/uapi -I ./include/uapi -I include/generated/uapi -D __KERNEL__ -D __HAVE_BUILTIN_BSWAP16__ -D __HAVE_BUILTIN_BSWAP32__ -D __HAVE_BUILTIN_BSWAP64__ -O2 -Wno-deprecated-declarations -Wno-gnu-variable-sized-type-not-at-end -Wno-pragma-once-outside-header -Wno-address-of-packed-member -Wno-unknown-warning-option -Wno-unused-value -Wno-pointer-sign -fdebug-compilation-dir /usr/src/linux-headers-4.4.0-112-generic -ferror-limit 19 -fmessage-length 168 -fobjc-runtime=gcc -fdiagnostics-show-option -vectorize-loops -vectorize-slp -o main.bc -x c /virtual/main.c

#include <linux/ptrace.h>
__attribute__((section(".bpf.fn.kprobe__tty_write")))
int kprobe__tty_write(struct pt_regs *ctx) { struct file *file = ctx->di; const char __user *buf = ctx->si; size_t count = ctx->dx;
return 0;
}

In the same manner, bcc extract the function name from kprobe__tty_write and automagically attaches the BPF program to tty_write.

Error while loading a BPF program that copies a buffer to the BPF stack

1 Answers