Compile Linux Kernel Modules into LLVM .bc bitcode

Question

Background

I'm trying to compile certain drivers within the Linux kernel: drm (drivers/gpu/drm/drm_drv.o) and radeon (drivers/gpu/drm/radeon/) gpu drivers. I'm using LLVM for the purposes of static analysis (tracking the arguments used in copy_to/from_user() invocations).

So far, I'm able to compile the actual modules using the Makefile as shown below:

make CC=clang CFLAGS=-emit-llvm drivers/gpu/drm/radeon/ 

But this does not actually emit any llvm bitcode -- I need the .bc files to run my pass with opt.

I only know how to generate .bc files when using clang directly (like below), but not with Makefiles...

clang -emit-llvm hello.c -c -o hello.bc 

Since that worked, I grabbed the verbose output of the GNU make operation, changed gcc to clang, and ran it to create the .bc file, which also worked:

 clang -emit-llvm  [[tons of CFLAGS]]  -c -o drm_drv.bc drivers/gpu/drm/drm_drv.c 

The only problem with that is I can only process a single C file in the kernel module at a time. Also it's very tedious to do this approach...

Main Problem

Which brings me to my main question: How would you go about emitting llvm .bc bitcode files using the kernel's Makefiles?

Or, if .bc bitcode creation must be done on a per-file basis, then how would I link them all together at the end so that I can run an LLVM opt pass on the aggregate of all the .bc files in a kernel module?

Answer 1

The best way to get LLVM IR in bitcode form out of Clang is to do LTO with the -flto command line flag.

If you have multiple translation units you can combine them together using the llvm-lto tool to "link" the bitcode files. Typically it generates code, but you can get it to drop the merged LLVM IR module with the -save-merged-module flag.

But none of this is really a supported interface. If this is a significantly useful workflow, you should talk to the LLVM developers about supporting something more akin to ld's -r.

Answer 2

I used clang to instrument ext2 module. There are 3 things that you want to do: 1) convert .c into .bc 2) run your optimiser on .bc file, and create an ext2-opt.o file 3) create ext2-instrumented.ko file from the ext2-opt.o file.

along with this, you also require a linux version compiled with clang. I could get linux 4.17 compiled with clang version 3.8.1 after disabling few modules. you can get it here

now, lets move to step 1 - as you said, run Make in verbose mode

make V=1 M=fs/ext2

and grab all options that are spit out by Makefile. the default compiler will be gcc. replace gcc with clang, add --emit-llvm command, and replace .o with .bc. the resultant compilataion command on my machine looks like so:

cd ../../ && clang -emit-llvm -Wp,-MD,fs/ext2/.all.o.d  -nostdinc -isystem /usr/local/bin/../lib/clang/3.8.1/include -I./arch/x86/include -Iarch/x86/include/generated/uapi -Iarch/x86/include/generated  -Iinclude -I./arch/x86/include/uapi -Iarch/x86/include/generated/uapi -I./include/uapi -Iinclude/generated/uapi -include ./include/linux/kconfig.h -D__KERNEL__ -Qunused-arguments -Wno-unknown-warning-option -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -std=gnu89 -no-integrated-as -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -m64 -mtune=generic -mno-red-zone -mcmodel=kernel -funit-at-a-time -DCONFIG_X86_X32_ABI -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -DCONFIG_AS_CFI_SECTIONS=1 -DCONFIG_AS_FXSAVEQ=1 -DCONFIG_AS_SSSE3=1 -DCONFIG_AS_CRC32=1 -DCONFIG_AS_AVX=1 -DCONFIG_AS_AVX2=1 -DCONFIG_AS_SHA1_NI=1 -DCONFIG_AS_SHA256_NI=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -O2 -Wframe-larger-than=1024 -fno-stack-protector -Wno-unused-variable -Wno-format-invalid-specifier -Wno-gnu -Wno-asm-operand-widths -Wno-initializer-overrides -fno-builtin -Wno-tautological-compare -mno-global-merge -fno-omit-frame-pointer -fno-optimize-sibling-calls -g -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -Werror=implicit-int -Werror=strict-prototypes -Werror=date-time -Wno-initializer-overrides -Wno-unused-value -Wno-format -Wno-unknown-warning-option -Wno-sign-compare -Wno-format-zero-length -Wno-uninitialized  -DMODULE  -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(all)"  -D"KBUILD_MODNAME=KBUILD_STR(ext2)" -mcmodel=kernel -c -o fs/ext2_instrumented/all.bc fs/ext2/all.c && cd -

this will create a .bc file for you. note that I have changed the resultant .bc codes directory to ext2_instrumented, i.e. my bc file is not created in fs/ext2 but in another folder called fs/ext2_instrumented. this is my work folder. I want this folder to not have any .c files, only .bc files. I need this because the default KBuild system looks for .c files in a folder. More on this later.

Step 2: Run all optimisation passes on your resultant ext2-instrumented.bc file using opt command like so:

opt -load $(DIR)/build/FSlice.so -constprop -sccp -mergereturn -sink -licm -reg2mem all.bc -o all.inst.bc llvm-link -o all.inst2.bc $(DIR)/build/libFSlice.bc all.inst.bc clang -mcmodel=kernel -c all.inst2.bc -o all.o

this will result into a .o file, which we will now compile into a .ko file in the next step:

Step 3:

make -C /lib/modules/$(shell uname -r)/build M=$(PWD) ext2instrumented.ko

Kbuild system is tough to understand, and even tougher to make modifications on. I would recommend use this 2 folder hack (1st folder to create .bc file and second folder to create .o and .ko files). To summarise, here is my resultant makefile in ext2_instrumented folder:

DIR=/home/fslice/fslice

obj-m += ext2instrumented.o

ext2instrumented-objs := all.o

all:
        cd ../../ && clang -emit-llvm -Wp,-MD,fs/ext2/.all.o.d  -nostdinc -isystem /usr/local/bin/../lib/clang/3.8.1/include -I./arch/x86/include -Iarch/x86/include/generated/uapi -Iarch/x86/include/generated  -Iinclude -I./arch/x86/include/uapi -Iarch/x86/include/generated/uapi -I./include/uapi -Iinclude/generated/uapi -include ./include/linux/kconfig.h -D__KERNEL__ -Qunused-arguments -Wno-unknown-warning-option -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -std=gnu89 -no-integrated-as -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -m64 -mtune=generic -mno-red-zone -mcmodel=kernel -funit-at-a-time -DCONFIG_X86_X32_ABI -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -DCONFIG_AS_CFI_SECTIONS=1 -DCONFIG_AS_FXSAVEQ=1 -DCONFIG_AS_SSSE3=1 -DCONFIG_AS_CRC32=1 -DCONFIG_AS_AVX=1 -DCONFIG_AS_AVX2=1 -DCONFIG_AS_SHA1_NI=1 -DCONFIG_AS_SHA256_NI=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -O2 -Wframe-larger-than=1024 -fno-stack-protector -Wno-unused-variable -Wno-format-invalid-specifier -Wno-gnu -Wno-asm-operand-widths -Wno-initializer-overrides -fno-builtin -Wno-tautological-compare -mno-global-merge -fno-omit-frame-pointer -fno-optimize-sibling-calls -g -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -Werror=implicit-int -Werror=strict-prototypes -Werror=date-time -Wno-initializer-overrides -Wno-unused-value -Wno-format -Wno-unknown-warning-option -Wno-sign-compare -Wno-format-zero-length -Wno-uninitialized  -DMODULE  -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(all)"  -D"KBUILD_MODNAME=KBUILD_STR(ext2)" -mcmodel=kernel -c -o fs/ext2_instrumented/all.bc fs/ext2/all.c && cd -
        opt -load $(DIR)/build/FSlice.so -constprop -sccp -mergereturn -sink -licm -reg2mem all.bc -o all.inst.bc
        llvm-link -o all.inst2.bc $(DIR)/build/libFSlice.bc all.inst.bc
        clang -mcmodel=kernel -c all.inst2.bc -o all.o
        make -C /lib/modules/$(shell uname -r)/build M=$(PWD) ext2instrumented.ko

clean:
        make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean
        rm -rf *.bc