TL;DR
Add -Wl,-Bsymbolic to the gcc linkage options for your shared library.
Why?
You're testing the PICness of h264_cabac.o with:
readelf --relocs h264_cabac.o | egrep '(GOT|PLT|JU?MP_SLOT)
and concluding the the object file was compiled with -fPIC if you get any
hits. Presumably you got this test from the favourite answer
to How can I tell, with something like objdump, if an object file has been built with -fPIC?
You got some hits, and I can reproduce that more than one way:
From the source code
$ git clone https://github.com/FFmpeg/FFmpeg.git
$ cd FFmpeg
$ ./configure --enable-shared
$ make
Then:
$ cd libavcodec
$ readelf --relocs h264_cabac.o | egrep '(GOT|PLT|JU?MP_SLOT)'
00000000175d 003100000004 R_X86_64_PLT32 0000000000000000 __stack_chk_fail - 4
000000001926 003100000004 R_X86_64_PLT32 0000000000000000 __stack_chk_fail - 4
00000000259f 003100000004 R_X86_64_PLT32 0000000000000000 __stack_chk_fail - 4
000000002f0d 003100000004 R_X86_64_PLT32 0000000000000000 __stack_chk_fail - 4
000000003216 003200000004 R_X86_64_PLT32 0000000000000000 av_log - 4
000000003460 00330000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_chroma422_dc_s - 4
000000003afc 003100000004 R_X86_64_PLT32 0000000000000000 __stack_chk_fail - 4
000000003fb6 00360000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_i_mb_type_info - 4
000000004031 00370000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_mb_sizes - 4
00000000409a 003800000004 R_X86_64_PLT32 0000000000000000 ff_init_cabac_decoder - 4
000000004248 00390000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_b_mb_type_info - 4
000000004299 003a00000004 R_X86_64_PLT32 0000000000000000 ff_h264_pred_direct_mo - 4
000000004a31 003b00000004 R_X86_64_PLT32 0000000000000000 ff_h264_check_intra4x4 - 4
000000004bd5 003200000004 R_X86_64_PLT32 0000000000000000 av_log - 4
000000004f85 003c0000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_p_mb_type_info - 4
0000000050fd 003d0000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_b_sub_mb_type_ - 4
000000005233 003a00000004 R_X86_64_PLT32 0000000000000000 ff_h264_pred_direct_mo - 4
00000000544a 003200000004 R_X86_64_PLT32 0000000000000000 av_log - 4
000000005bef 003a00000004 R_X86_64_PLT32 0000000000000000 ff_h264_pred_direct_mo - 4
000000006db5 003e00000004 R_X86_64_PLT32 0000000000000000 ff_h264_check_intra_pr - 4
000000006de9 003f0000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_p_sub_mb_type_ - 4
000000007171 003200000004 R_X86_64_PLT32 0000000000000000 av_log - 4
000000008b1b 003e00000004 R_X86_64_PLT32 0000000000000000 ff_h264_check_intra_pr - 4
00000000ad41 004000000009 R_X86_64_GOTPCREL 0000000000000000 ff_h264_chroma_dc_scan - 4
00000000ad84 004000000009 R_X86_64_GOTPCREL 0000000000000000 ff_h264_chroma_dc_scan - 4
00000000b758 003100000004 R_X86_64_PLT32 0000000000000000 __stack_chk_fail - 4
From the Ubuntu 16.04 dev package
$ sudo apt-get install libavcodec-dev
$ dpkg -S libavcodec.a
libavcodec-dev:amd64: /usr/lib/x86_64-linux-gnu/libavcodec.a
$ mkdir ~/deleteme
$ cd ~/deleteme
$ ar x /usr/lib/x86_64-linux-gnu/libavcodec.a h264_cabac.o
$ readelf --relocs h264_cabac.o | egrep '(GOT|PLT|JU?MP_SLOT)'
0000000000c7 002e00000004 R_X86_64_PLT32 0000000000000000 __stack_chk_fail - 4
0000000002fa 002e00000004 R_X86_64_PLT32 0000000000000000 __stack_chk_fail - 4
00000000179d 002e00000004 R_X86_64_PLT32 0000000000000000 __stack_chk_fail - 4
000000001966 002e00000004 R_X86_64_PLT32 0000000000000000 __stack_chk_fail - 4
000000001b09 002e00000004 R_X86_64_PLT32 0000000000000000 __stack_chk_fail - 4
000000001d4a 002e00000004 R_X86_64_PLT32 0000000000000000 __stack_chk_fail - 4
000000001ee5 002e00000004 R_X86_64_PLT32 0000000000000000 __stack_chk_fail - 4
00000000265f 002e00000004 R_X86_64_PLT32 0000000000000000 __stack_chk_fail - 4
000000002fcd 002e00000004 R_X86_64_PLT32 0000000000000000 __stack_chk_fail - 4
0000000032f6 002f00000004 R_X86_64_PLT32 0000000000000000 av_log - 4
000000003305 002e00000004 R_X86_64_PLT32 0000000000000000 __stack_chk_fail - 4
000000003bdc 002e00000004 R_X86_64_PLT32 0000000000000000 __stack_chk_fail - 4
000000003cb5 002e00000004 R_X86_64_PLT32 0000000000000000 __stack_chk_fail - 4
000000004121 00320000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_mb_sizes - 4
000000004187 003300000004 R_X86_64_PLT32 0000000000000000 ff_init_cabac_decoder - 4
000000004381 003400000004 R_X86_64_PLT32 0000000000000000 ff_h264_pred_direct_mo - 4
000000004afe 003500000004 R_X86_64_PLT32 0000000000000000 ff_h264_check_intra4x4 - 4
000000005556 003400000004 R_X86_64_PLT32 0000000000000000 ff_h264_pred_direct_mo - 4
00000000576a 002f00000004 R_X86_64_PLT32 0000000000000000 av_log - 4
000000005acf 003400000004 R_X86_64_PLT32 0000000000000000 ff_h264_pred_direct_mo - 4
000000006e31 002f00000004 R_X86_64_PLT32 0000000000000000 av_log - 4
000000006e58 003600000004 R_X86_64_PLT32 0000000000000000 ff_h264_check_intra_pr - 4
000000009c20 003600000004 R_X86_64_PLT32 0000000000000000 ff_h264_check_intra_pr - 4
00000000b425 002f00000004 R_X86_64_PLT32 0000000000000000 av_log - 4
00000000b5ab 002e00000004 R_X86_64_PLT32 0000000000000000 __stack_chk_fail - 4
The results aren't identical, and I get 26 relocations the first way, 25 the second way. But there are plenty of PIC-safe relocations either way and
I'm happy to believe both compilations of h264_cabac.o had -fPIC, whatever other options they had.
I'll state the obvious: The symbol ff_h264_cabac_tables, about which your linkage
complains:
relocation R_X86_64_PC32 against symbol 'ff_h264_cabac_tables' can not be used when making a shared object
isn't in either of those lists. That means this object file - from both provenances - contains both PIC-safe and PIC-unsafe relocations.
How could GCC get that wrong, without anybody noticing till now? And if it did, how
did I just run the shared library build of FFmpeg and link libavcodec.so successfully?
Let's have a look at the PIC-unsafe relocations:
$ readelf --relocs h264_cabac.o | egrep -v '(GOT|PLT|JU?MP_SLOT)'
000000000017 002c00000002 R_X86_64_PC32 0000000000000000 ff_h264_cabac_tables - 4
...
...
Well I'll elide ~160 lines of that, but they all describe PC-relative type R_X86_64_PC32
relocations and the only symbol mentioned, discounting section names and local labels,
is our friend ff_h264_cabac_tables, about which the symbol table says:
$ readelf -s h264_cabac.o | grep ff_h264_cabac_tables
44: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND ff_h264_cabac_tables
It's a global variable, not defined in this object file.
GCC's -fPIC isn't broken. However, knowing that an object file was compiled with
-fPIC can't absolutely guarantee that it contains no PC-relative type
R_X86_64_PC32 relocations that reference undefined global symbols. relocation R_X86_64_PC32 against symbol 'ff_h264_cabac_tables' is
such a relocation. The R_X86_64_PC32 type relocation employs a 32-bit PC-relative addressing mode,
which is efficient but has a critical limitation in the setting of a 64-bit linkage.
The linker can't guarantee the referenced symbol will not be dynamically resolved
to an address that's unrepresentable in this addressing mode. It won't have that, so
it says:
relocation R_X86_64_PC32 against symbol 'ff_h264_cabac_tables' can not be used when making a shared object
and its advice:
recompile with -fPIC
is based on the hypothesis that the culprit object file was not compiled with -fPIC.
Which is probably, but not necessarily, the right hypothesis and is not right about
your culprit libavcodec.a(h264_cabac.o)
Compiling with -fPIC will guarantee you PIC-safe relocations provided that the compiler
is allowed to do all the assembly and code-generation. But it wasn't allowed to
with your specimen of h264_cabac.o or either of my specimens. All those specimens
were compiled from FFmpeg/libavcodec/x86/h264_cabac.c in the FFmpeg source tree.
Look at that file and you'll see that it defines functions that refer to the extern
global variable ff_h264_cabac_tables and are implemented in inline, hand-crafted
assembly. GCC can be told to compile these functions -fPIC, but it doesn't get
a chance. The position-independence of these functions is the responsibility of
the author of the assembly code.
We can show that GCC is able to compile a h264_cabac.o that has only PIC-safe
relocations, if it's allowed to. That will collaterally prove that your linkage
failure stems from the hand-assembly of our specimens of the file and will also show
you one fix for the linkage failure. FFmpeg's ./configure script has the option:
--disable-asm disable all assembly optimizations
which has the effect, among others, of causing h264_cabac.o to be compiled
from the pure C source file FFmpeg/libavcodec/h264_cabac.c instead of the the
inline assembly source FFmpeg/libavcodec/x86/h264_cabac.c. So let's try that:
$ cd FFmpeg
$ make clean
$ ./configure --enable-shared --disable-asm
$ make
$ cd libavcodec
$ readelf --relocs h264_cabac.o | grep ff_h264_cabac_tables
00000000000a 00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
0000000000ca 00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
000000001eb5 00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
0000000021c6 00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
0000000026fe 00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
000000002a17 00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
000000002f13 00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
00000000324c 00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
000000003509 00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
00000000362a 00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
0000000037d7 00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
00000000592b 00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
Now, all the relocations referencing ff_h264_cabac_tables are PIC-safe.
We might as well prove that this h264_cabac.o can be linked in a shared library. We know
that ff_h264_cabac_tables is undefined in h264_cabac.o, so we'll also need to link the
object file in which it is defined. It happens to be ./cabac.o.
$ gcc -shared -o libfoo.so h264_cabac.o cabac.o
Voila:
$ file libfoo.so
libfoo.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=ed63107b715b357853da94d4a031c0b06c30c5f2, not stripped
You might still feel a little aggrieved, however, if you have to link your own
shared library with this unoptimized h264_cabac.o and a little disappointed by
the assembly coding flaw that obliges you to. These feelings would be premature.
Remember, I've already built FFmpeg successfully with plain vanilla ./configure --enable-shared.
I said that linker's objection in your failing linkage is that the R_X86_64_PC32
relocation referencing ff_h264_cabac_tables might not be feasible at runtime, if ff_h264_cabac_tables
is dynamically resolved. It is not an objection to the type R_X86_64_PC32
relocation as such. It is a precautionary objection, based on ignorance of how ff_h264_cabac_tables
will ultimately be resolved.
But we know that ff_h264_cabac_tables is in fact defined in cabac.o and that
we'll include it the same linkage with h264_cabac.o, just as they're both included
in the linkage of libavcodec.so. And we can tell the linker that any global
references in the linkage are to be statically resolved to definitions in the
shared library being linked, if at all, by passing it the argument:
-Bsymbolic
This will remove its precautionary objection to any R_X86_64_PC32 relocation.
It knows it will be able to decide at linktime whether the R_X86_64_PC32
relocation against ff_h264_cabac_tables is feasible. If not, it will give
a different error: relocation truncated to fit:... Otherwise it will succeed
without comment.
Inevitably, that's how libavcodec.so is successfully linked in the stock FFmpeg build.
Once more from the top:
$ cd FFmpeg
$ make clean
$ ./configure --enable-shared
$ make
Then force a relink of libavcodec.so:
$ rm libavcodec/h264_cabac.o
$ $ make libavcodec/libavcodec.so V=1
gcc -I. -I./ -D_ISOC99_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE \
-D_POSIX_C_SOURCE=200112 -D_XOPEN_SOURCE=600 -DPIC -DZLIB_CONST -DHAVE_AV_CONFIG_H \
-std=c11 -fomit-frame-pointer -fPIC -pthread -g -Wdeclaration-after-statement \
-Wall -Wdisabled-optimization -Wpointer-arith -Wredundant-decls -Wwrite-strings \
-Wtype-limits -Wundef -Wmissing-prototypes -Wno-pointer-to-int-cast -Wstrict-prototypes \
-Wempty-body -Wno-parentheses -Wno-switch -Wno-format-zero-length -Wno-pointer-sign \
-O3 -fno-math-errno -fno-signed-zeros -fno-tree-vectorize -Werror=format-security \
-Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=return-type \
-Werror=vla -Wformat -fdiagnostics-color=auto -Wno-maybe-uninitialized \
-MMD -MF libavcodec/h264_cabac.d -MT libavcodec/h264_cabac.o -c \
-o libavcodec/h264_cabac.o libavcodec/h264_cabac.c
sed 's/MAJOR/57/' libavcodec/libavcodec.v | cat > libavcodec/libavcodec.ver
gcc -shared -Wl,-soname,libavcodec.so.57 -Wl,-Bsymbolic ... etc. etc. ...
^^^^^^^^^^^^^^
So there is no assembly coding flaw. To link the hand-optimized h264_cabac.o
in a shared library you just need to add -Wl,-Bsymbolic to the gcc linkage options. It's
a requirement of the optimization.
Let's prove that minimally:
$ cd libavcodec/
$ gcc -shared -o libfoo.so h264_cabac.o cabac.o
/usr/bin/ld: h264_cabac.o: relocation R_X86_64_PC32 against symbol `ff_h264_cabac_tables' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
There's your failure again. And:
$ gcc -shared -Wl,-Bsymbolic -o libfoo.so h264_cabac.o cabac.o
$ file libfoo.so
libfoo.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=7dc86aeae353c4d92cdb5fa35d169bf019b47eb2, not stripped
success.
